This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 257563

Summary: surefire 2.19.1 show no test results windows
Product: projects Reporter: skygo <skygo>
Component: MavenAssignee: markiewb
Status: VERIFIED FIXED    
Severity: normal CC: karileh, markiewb, radzikowski, Saljack, Yaytay
Priority: P2 Keywords: NETFIX, PATCH_AVAILABLE
Version: 8.2   
Hardware: PC   
OS: Windows 10 x64   
See Also: https://issues.apache.org/jira/browse/SUREFIRE-1158
https://issues.apache.org/jira/browse/SUREFIRE-1216
Issue Type: DEFECT Exception Reporter:
Attachments: Snapshot of POM file
Sample project with surefire 2.19.2-SN
Patch to support surefire >=2.19

Description skygo 2016-01-14 13:45:58 UTC
According to issue : https://netbeans.org/bugzilla/show_bug.cgi?id=256211 surefire 2.19.1 make test possible on a maven build

But the test phase is now partialy broken as the Test Results windows is not shown.

Netbeans maven.junit use the log to "detect" surefire report
but with surefire 2.19 see [1] the log is not showing the following sentence:
Surefire report directory 

Output:
V2.18:
--- maven-surefire-plugin:2.18:test (default-test) @ project ---
Surefire report directory: c:\dummyfolder\target\surefire-reports
EOF
V2.19.1
--- maven-surefire-plugin:2.19.1:test (default-test) @ project ---
EOF

A possible user side fix is to add -e in the related test actions
 (test,test.single,debug.test.single)
 
[1] 
https://issues.apache.org/jira/browse/SUREFIRE-1158
Comment 1 mitya 2016-02-24 20:13:31 UTC
Same here, NetBeans 8.1 with latest updates, Oracle JDK 8, Linux 32-bit. Have to stick to maven-surefire-plugin 2.18.

Probably screenscraping plugin output is a bad idea, can it be done in some other way?
Comment 2 flynnk 2016-03-08 03:34:04 UTC
Just spent 30 minutes hunting down this bug while setting up a new project.  I think this needs to be raised in priority, just because getting the 'green bar' in the test results window is such a key part of the workflow, you get a vague sense of panic when it does not appear.  There is nothing in the GUI to help you debug this, either--no error message or anything.
Comment 3 helmut.at.work 2016-03-15 07:38:12 UTC
Same on my machine. NB 8.1 surefire 2.18.1 work and surefire 2.19 doesn't.
Found one difference in the surefire output.
In 2.19 output there is the namespace defined in the  testsuite tag.

<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name=...

In 2.18.1 not

<testsuite name=...

Maybe it will help .
Comment 4 etrof 2016-03-24 19:20:26 UTC
> I think this needs to be raised in priority

Agreed, without the test result it's really annoying.
Comment 5 Tomas Stupka 2016-05-17 13:15:07 UTC
*** Bug 259108 has been marked as a duplicate of this bug. ***
Comment 6 nedenom 2016-05-18 15:44:23 UTC
This is almost a blocker for me. I can't even see the stacktraces when tests are in error. 

I am the only one on my team using Netbeans, do you think the others, the Eclipse and IDEA users, are willing to downgrade surefire just because my Netbeans don't cut it?
Comment 7 javydreamercsw 2016-05-19 14:04:18 UTC
(In reply to nedenom from comment #6)
> This is almost a blocker for me. I can't even see the stacktraces when tests
> are in error. 
> 
> I am the only one on my team using Netbeans, do you think the others, the
> Eclipse and IDEA users, are willing to downgrade surefire just because my
> Netbeans don't cut it?

Just downgrade to 2.18.1. Looks to be a bug on surefire.
Comment 8 jmborer 2016-05-19 14:18:50 UTC
(In reply to javydreamercsw from comment #7)
> (In reply to nedenom from comment #6)
> > This is almost a blocker for me. I can't even see the stacktraces when tests
> > are in error. 
> > 
> > I am the only one on my team using Netbeans, do you think the others, the
> > Eclipse and IDEA users, are willing to downgrade surefire just because my
> > Netbeans don't cut it?
> 
> Just downgrade to 2.18.1. Looks to be a bug on surefire.

Did you look at helmut's hint:

In 2.19.1

<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name=...

In 2.18.1 not

<testsuite name=...

If it is that the reason, then I would'nt say it is Surefire bug.
Comment 9 aede 2016-08-10 07:46:16 UTC
(In reply to nedenom from comment #6)
> This is almost a blocker for me. I can't even see the stacktraces when tests
> are in error. 
> 
> I am the only one on my team using Netbeans, do you think the others, the
> Eclipse and IDEA users, are willing to downgrade surefire just because my
> Netbeans don't cut it?

Same here... I am the only one using Netbeans and working on a code base together with some 25 other developers.

This bug should effect every user that uses maven projects and should be addressed.
Comment 10 hlo 2016-08-11 20:05:39 UTC
Created attachment 161643 [details]
Snapshot of POM file
Comment 11 hlo 2016-08-11 20:13:57 UTC
The bug is still there. 
Instead of setting <skipTests> to false just comment out the thing, build with dependencies, and then your ready to go.  
Does it help to downgrade SureFIre plugin if running NB 8.1?
Comment 12 markiewb 2016-08-25 21:46:06 UTC
It is a known surefire 2.19.1 bug. https://issues.apache.org/jira/browse/SUREFIRE-1216

Wait for 2.19.2!
Comment 13 jmborer 2016-08-26 07:56:46 UTC
OK interesting. So finally it is Surefire bug. Can't wait for 2.19.2. 

Thank you Benno for investigating.
Comment 14 skygo 2016-09-06 11:27:06 UTC
Hi, 
  I tried the 2.19.2-SNAPSHOT, Test Results window show again but in case of failure the stack trace is not present, it only shows the last error.

Not sure but it may need adaptation on NetBeans side maybe.
Comment 15 markiewb 2016-09-06 20:34:21 UTC
Created attachment 161931 [details]
Sample project with surefire 2.19.2-SN
Comment 16 markiewb 2016-09-06 22:04:23 UTC
(In reply to skygo from comment #14)
> Hi, 
>   I tried the 2.19.2-SNAPSHOT, Test Results window show again but in case of
> failure the stack trace is not present, it only shows the last error.
> 
> Not sure but it may need adaptation on NetBeans side maybe.

Yes, it broken because surefire changed their output format. No even the testoutput window is opened. Try out the maven project attached at https://netbeans.org/bugzilla/attachment.cgi?id=161931

Here are the outputs
====%<===============================================================================
--- maven-surefire-plugin:2.18:test (default-test) @ mavenproject1 ---
Surefire report directory: C:\Users\markiewb\AppData\Local\Temp\mavenproject1\target\surefire-reports
file.encoding cannot be set as system property, use <argLine>-Dfile.encoding=...<argLine> instead

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running test.mavenproject1.NewClassTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in test.mavenproject1.NewClassTest

Results :




Tests run: 1, Failures: 0, Errors: 0, Skipped: 0


--- maven-surefire-plugin:2.19.2-SNAPSHOT:test (default-test) @ mavenproject1 ---
file.encoding cannot be set as system property, use <argLine>-Dfile.encoding=...</argLine> instead

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running test.mavenproject1.NewClassTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in test.mavenproject1.NewClassTest

Results:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

====%<===============================================================================
Since in 2.19.x the line 
Surefire report directory: C:\Users\markiewb\AppData\Local\Temp\mavenproject1\target\surefire-reports
is missing. 

But this line is required to get the current project in
org.netbeans.modules.maven.junit.JUnitOutputListenerProvider.createSession(File)
        outDirPattern = Pattern.compile ("(?:\\[INFO\\] )?Surefire report directory\\: (?<" + GROUP_FILE_NAME + ">.*)", Pattern.DOTALL); //NOI18N
        outDirPattern2 = Pattern.compile("(?:\\[INFO\\] )?Setting reports dir\\: (?<" + GROUP_FILE_NAME + ">.*)", Pattern.DOTALL); //NOI18N

It has been removed by https://issues.apache.org/jira/browse/SUREFIRE-1158 https://github.com/apache/maven-surefire/pull/95/commits/67e6606d51e24ecfd4bfe75e4f4dd906f9e88435 by decreasing the log level.
Comment 17 markiewb 2016-09-06 22:30:19 UTC
Created attachment 161934 [details]
Patch to support surefire >=2.19

Here is a patch, which reads the output directory from the pom configuration instead from the console.

@NB-Dev: Please review! It works with surefire 2.19, 2.19.1, 2.19.2-SN and 2.20-SN
Comment 18 skygo 2016-09-07 10:10:21 UTC
Works well and also backward compatible with 2.18 :p

Only stacktrace is different in case of error

if you alter NewClass.method to make it public and have the following body
throw new UnsupportedOperationException("ISSUE."); 

if you alter NewClassTest.testMain to have the following body
NewClass newClass = new NewClass();
newClass.method();

With surefire 2.18
java.lang.UnsupportedOperationException: ISSUE.
	at test.mavenproject1.NewClass.method(NewClass.java:22)
	at test.mavenproject1.NewClassTest.testMain(NewClassTest.java:23)

With surefire 2.19.2
java.lang.UnsupportedOperationException: ISSUE.
	at test.mavenproject1.NewClassTest.testMain(NewClassTest.java:23)

I cannot figure out what change in surefire for this issue, but this is annoying because it's prevent better usage we can do with NetBeans to track the root cause of the exception to correct class if test seems ok.
Comment 19 Jiri Kovalsky 2016-09-07 20:12:02 UTC
With 13 votes this issue needs to be P2 according to our Bug Priority Guidelines [1]. Updating accordingly.

[1] http://wiki.netbeans.org/BugPriorityGuidelines
Comment 20 markiewb 2016-09-07 20:52:33 UTC
(In reply to skygo from comment #18)
> Works well and also backward compatible with 2.18 :p
> 
> Only stacktrace is different in case of error
> 
> if you alter NewClass.method to make it public and have the following body
> throw new UnsupportedOperationException("ISSUE."); 
> 
> if you alter NewClassTest.testMain to have the following body
> NewClass newClass = new NewClass();
> newClass.method();
> 
> With surefire 2.18
> java.lang.UnsupportedOperationException: ISSUE.
> 	at test.mavenproject1.NewClass.method(NewClass.java:22)
> 	at test.mavenproject1.NewClassTest.testMain(NewClassTest.java:23)
> 
> With surefire 2.19.2
> java.lang.UnsupportedOperationException: ISSUE.
> 	at test.mavenproject1.NewClassTest.testMain(NewClassTest.java:23)
> 
> I cannot figure out what change in surefire for this issue, but this is
> annoying because it's prevent better usage we can do with NetBeans to track
> the root cause of the exception to correct class if test seems ok.

-> Please file a new issue for that. At NB bugzilla and at the Surefire-Plugin-JIRA, because trimming the stacktrace by surefire 2.19 removes relevant information.

I guess it is caused by https://issues.apache.org/jira/browse/SUREFIRE-1067 / https://issues.apache.org/jira/browse/SUREFIRE-1050

As a workaround use -DtrimStackTrace=false
Comment 21 Tomas Stupka 2016-09-08 09:32:11 UTC
> Created attachment 161934 [details]
> Patch to support surefire >=2.19
> Here is a patch, which reads the output directory from the pom configuration instead from the console.
looks ok, feel free to integrate

thanks
Comment 22 Tomas Stupka 2016-09-08 10:07:43 UTC
fixed in jet-main #8376384b90a3
Comment 23 Tomas Stupka 2016-09-08 10:29:08 UTC
integrated into releases82 #ff862e19c5a7
Comment 24 Quality Engineering 2016-09-09 01:52:54 UTC
Integrated into 'main-silver', will be available in build *201609090002* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/8376384b90a3
User: Tomas Stupka <tstupka@netbeans.org>
Log: Issue #257563 - surefire 2.19.1 show no test results windows
Comment 25 Jiri Kovalsky 2016-09-09 10:11:48 UTC
Eric, can you please verify this bug? Thanks!
Comment 26 skygo 2016-09-09 11:16:49 UTC
Tests Windows is now working with 2.19.1.

To get the full stack as it was before you need to set the parameter trimStackTrace to false (not a workaround but the legit way :p)

https://issues.apache.org/jira/browse/SUREFIRE-1277

Thanks for the fix !
Comment 27 javydreamercsw 2017-03-07 17:07:30 UTC
Was this released on 8.2? I still see this issue. It is marked as 8.2 milestone and has been fixed for almost 6 months already.