This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 207886 - Cannot run TestNG tests in Ant project
Summary: Cannot run TestNG tests in Ant project
Status: RESOLVED FIXED
Alias: None
Product: java
Classification: Unclassified
Component: TestNG (show other bugs)
Version: 7.1
Hardware: PC Windows 7
: P1 normal (vote)
Assignee: Lukas Jungmann
URL:
Keywords:
Depends on:
Blocks: 130605
  Show dependency tree
 
Reported: 2012-01-29 19:52 UTC by _ gtzabari
Modified: 2012-03-22 10:53 UTC (History)
0 users

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
nbtargets-testng.xml updated for testng 6.3.1 (11.86 KB, text/xml)
2012-01-29 21:12 UTC, _ gtzabari
Details

Note You need to log in before you can comment on or make changes to this bug.
Description _ gtzabari 2012-01-29 19:52:09 UTC
Product Version: NetBeans IDE Dev (Build 201201250600)
Java: 1.7.0_02; Java HotSpot(TM) 64-Bit Server VM 22.0-b10
System: Windows 7 version 6.1 running on amd64; Cp1252; en_US (nb)
User directory: C:\Users\Gili\.netbeans\dev
Cache directory: C:\Users\Gili\.netbeans\dev\var\cache

1. Open an existing project containing TestNG tests.
2. Install the TestNG Ant plugin 1.1.
3. ALT+F6
4. Netbeans fails because it is looking for JUnit tests instead of TestNG:

Testcase: initializationError(org.pill.repository.local.LocalRepositoryTest):	Caused an ERROR
No runnable methods
java.lang.Exception: No runnable methods
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)

This behavior contradicts http://wiki.netbeans.org/TestNG and bug 130605. Netbeans 7.1 does *not* support TestNG.
Comment 1 _ gtzabari 2012-01-29 20:36:18 UTC
I got a bit further by adding the following to build.xml:

	<target name="-post-compile-test">
		<testng verbose="2">
			<classpath path="${run.test.classpath}"/>
			<classfileset dir="build/test/classes" includes="**/*Test.class"/>
		</testng>
	</target>

But now I get two tabs under Test Results:

1. The "Ant Suite" tab reads "No tests executed"
2. The second tab has the name of the project. It contains the following tree:

  No test passed, 1 test failed
  \-> <test class name> FAILED
    \-> Warning FAILED

As you can see, it fails to provide any useful information about the failure.
Comment 2 _ gtzabari 2012-01-29 20:37:53 UTC
I forgot to mention, I also had to add the TestNG library to my project and add the following above -post-compile-test:

	<taskdef resource="testngtasks" classpath="lib/testng/testng-6.3.1.jar"/>

All of this should be handled automatically by installing the plugin or documented in great detail.
Comment 3 _ gtzabari 2012-01-29 21:12:03 UTC
I got rid of the second tab (the one containing WARNING) by replacing 

-post-compile-test with -do-test-run

Now the "Test Results" contains a single "Ant suite" tab. The test output shows up on the right (good!) but the left hand side still reads "0% - No tests executed."

I found an excellent article at http://www.oocities.org/nimarukan/netbeans/nbtargets/nbtargets-alt-test-runners.html that describes how to integrate TestNG into Netbeans. I'm going to attach an updated version of nbtargets-testng.xml that worked for me (but again, the "Test Results" tab claims no tests were executed).

At this point I'm looking for some feedback: how do I get the Test Results tab to work?
Comment 4 _ gtzabari 2012-01-29 21:12:48 UTC
Created attachment 115363 [details]
nbtargets-testng.xml updated for testng 6.3.1
Comment 5 Lukas Jungmann 2012-01-29 22:31:26 UTC
which version of the plugin do you use? the one from development UC or from stable UC for earlier NB version?

Is compile on save enabled or disabled for your project? The former case is not supported yet and there's a RFE for this already and some parts may still not work properly. The latter case should work - any help with hunting bugs here would be really appreciated



FYI: The plugin on dev UC contains TestNG 6.3.2beta (also note this is a slightly different version from the one provided by Cedric), so the version you used is really important info ... w/o that => incomplete (for now)
Comment 6 _ gtzabari 2012-01-29 23:31:21 UTC
(In reply to comment #5)
> which version of the plugin do you use? the one from development UC or from
> stable UC for earlier NB version?

I'm using "Ant TestNG support 1.1" from "Latest Development Build" (note I am using a NB 7.2 dev build).

> Is compile on save enabled or disabled for your project? The former case is not
> supported yet and there's a RFE for this already and some parts may still not
> work properly. The latter case should work - any help with hunting bugs here
> would be really appreciated

Compile on Save is disabled for my project.

> FYI: The plugin on dev UC contains TestNG 6.3.2beta (also note this is a
> slightly different version from the one provided by Cedric), so the version you
> used is really important info ... w/o that => incomplete (for now)

I manually added TestNG 6.3.1 to the "Test Libraries". Could that break anything? Should I remove it?

Assuming that I configure everything properly, should the Test Results window display my tests or should it show "No tests executed" as I mentioned?
Comment 7 _ gtzabari 2012-02-01 17:23:54 UTC
Lukas,

Can you reproduce this problem?
Comment 8 Lukas Jungmann 2012-02-01 17:41:23 UTC
sorry, haven't had a chance to try to reproduce this yet :-( But I'll have to resolve this by Monday, so if some fix is necessary, you can expected it during Fri/Sat.

One more question: is it java se project where you see this issue or is it another project type?

as for test results - does setting 'useDefaultListeners="true"' make any difference? Or can you try to refresh the view using filtering buttons? I'm aware there's some sync issue but after refresh, you should be able to see the stacktrace etc
Comment 9 _ gtzabari 2012-02-01 17:43:13 UTC
(In reply to comment #8)
> sorry, haven't had a chance to try to reproduce this yet :-( But I'll have to
> resolve this by Monday, so if some fix is necessary, you can expected it during
> Fri/Sat.

Excellent :)

> One more question: is it java se project where you see this issue or is it
> another project type?

Yes, it's a JavaSE project.

> as for test results - does setting 'useDefaultListeners="true"' make any
> difference? Or can you try to refresh the view using filtering buttons? I'm
> aware there's some sync issue but after refresh, you should be able to see the
> stacktrace etc

Where do I set useDefaultListeners?
Comment 10 Lukas Jungmann 2012-02-01 18:29:02 UTC
> > as for test results - does setting 'useDefaultListeners="true"' make any
> > difference? Or can you try to refresh the view using filtering buttons? I'm
> > aware there's some sync issue but after refresh, you should be able to see the
> > stacktrace etc
> 
> Where do I set useDefaultListeners?

<testng verbose="2" useDefaultListeners="true">

it's an attribute of testng task - you can verify that it is applied by running ant in verbose mode (see Tools -> Options-> Misc -> Ant -> Verbosity level=Verbose) - you should see a lot of messages starting with 'VerboseReporter' or sth like that
Comment 11 _ gtzabari 2012-02-03 04:27:33 UTC
I figured it out. It looks like the plugin fails silently when interacting with different versions of TestNG it was compiled against.

If you use TestNG 6.3.1 in your project, the results window will not work. If you use 6.3.2beta which ships with the plugin it'll work fine. You don't need to use "useDefaultListeners" either.

I think the biggest thing you're missing now is good documentation. I've added a "Getting Started" section at http://wiki.netbeans.org/TestNG#Getting_Started:_Ant_plugin
Comment 12 Lukas Jungmann 2012-02-03 11:01:41 UTC
first of all - thanks for the time you're spending with this.


(In reply to comment #11)
> I figured it out. It looks like the plugin fails silently when interacting with
> different versions of TestNG it was compiled against.
> 
> If you use TestNG 6.3.1 in your project, the results window will not work. If
> you use 6.3.2beta which ships with the plugin it'll work fine. You don't need
> to use "useDefaultListeners" either.

clearly a regression, which I thought I already fixed. The cause of it is that in previous versions all results were presented in the UI after all tests finished and in current version results are presented as they come after each test/testcase

> 
> I think the biggest thing you're missing now is good documentation. I've added
> a "Getting Started" section at
> http://wiki.netbeans.org/TestNG#Getting_Started:_Ant_plugin

Thanks! I'll try to improve it ASAP.
Comment 13 _ gtzabari 2012-02-03 14:17:06 UTC
(In reply to comment #12)
> clearly a regression, which I thought I already fixed. The cause of it is that
> in previous versions all results were presented in the UI after all tests
> finished and in current version results are presented as they come after each
> test/testcase

I get the feeling that you'd have a hard time unit testing the unit tests :) If it is possible, you might try using http://code.google.com/p/fest/ to avoid such regressions in the future.

Thanks for the support! I've been wanting to migrate to TestNG for a while now :)
Comment 14 Lukas Jungmann 2012-02-05 01:40:25 UTC
http://hg.netbeans.org/main/contrib/rev/becd301cfcbf should help

-as for build.xml modifications - this is sth what I simply cannot do from within the plugin (technically speaking I can but...)
Comment 15 _ gtzabari 2012-02-05 03:18:27 UTC
(In reply to comment #14)
> -as for build.xml modifications - this is sth what I simply cannot do from
> within the plugin (technically speaking I can but...)

Sorry, I didn't really understand what you meant about build.xml. Can you elaborate?
Comment 16 Lukas Jungmann 2012-02-05 11:29:04 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > -as for build.xml modifications - this is sth what I simply cannot do from
> > within the plugin (technically speaking I can but...)
> 
> Sorry, I didn't really understand what you meant about build.xml. Can you
> elaborate?

the file you attached comment #4 - overriding of target '-debug-start-debuggee-test' defined in build-impl.xml will work if it done in project's build.xml but it won't work if I do it in the customized build script - testng-build.xml - which is included from build-impl.xml
Comment 17 _ gtzabari 2012-02-05 14:53:42 UTC
(In reply to comment #16)
> the file you attached comment #4 - overriding of target
> '-debug-start-debuggee-test' defined in build-impl.xml will work if it done in
> project's build.xml but it won't work if I do it in the customized build script
> - testng-build.xml - which is included from build-impl.xml

Some features, like the Netbeans profiler, changes build.xml directly (adding an <include> statement for its sub-script). At the very least, you should offer your own testng-build.xml implementation and ask users to include it in their build.xml.
Comment 18 Lukas Jungmann 2012-02-05 16:38:45 UTC
(In reply to comment #17)
> Some features, like the Netbeans profiler, changes build.xml directly (adding
> an <include> statement for its sub-script).

they most likely have a very good reason why they are doing it

 At the very least, you should offer
> your own testng-build.xml implementation and ask users to include it in their
> build.xml.

there already is nbproject/testng-impl.xml (included from build-impl.xml) - to get it you have to create at least one test or test suite using new file wizard while not having testng lib on the cp but if you already have a project which has TestNG on the cp the file is not created (based on an assumption that if a project has the lib on the cp then it either already has the script present or it is some older or custom project where user added the lib himself and therefore he knows what he is doing - ie he already modified build.xml or included some other file with testng task definition etc
Comment 19 Lukas Jungmann 2012-02-05 23:36:35 UTC
(In reply to comment #18)
> there already is nbproject/testng-impl.xml 

http://hg.netbeans.org/main/contrib/rev/b4619fb1e7af should fix debugging related issues if testng-impl.xml is included directly from build.xml

Only note 2 things:
-compile on save must be disabled for global IDE actions and shortcuts to work as expected
-to get the build script working with arbitrary official testng distro you have to remove 'mixed' from java and testng tasks there
Comment 20 _ gtzabari 2012-02-06 14:59:15 UTC
If you don't mind, I'd like to reopen this issue until we resolve the basics of getting this working in an Ant project.

I took my project with existing TestNG tests, removed TestNG from the classpath and created a test using the New File wizard. build.xml was modified as you mentioned and TestNG was added to the classpath. I then removed the class I added using the New File wizard (only my old tests remained) and when I tried running with ALT+F6 I got:

[TestNGAntTask] TESTNG PASSED @C:\Users\Gili\AppData\Local\Temp\testng8168646004527825901 WHICH CONTAINS:
  -log
  2
  -usedefaultlisteners
  true
  -d
  C:\Users\Gili\Documents\pill\core\build\test\results
  -listener
  org.testng.reporters.VerboseReporter
  -suitename
  Ant suite
  -testname
  Ant test
Usage: <main class> [options]
You need to specify at least one testng.xml, one class or one method

I am expecting my existing tests to run. I don't think you can count on users using the file dialog for everything (I routinely create a new Java Class and turn it into a test class later on). Instead, I recommend you scan "Test Packages" for the @Test annotation. Let me know what you think.

> (In reply to comment #18)
> -to get the build script working with arbitrary official testng distro you have
> to remove 'mixed' from java and testng tasks there

Will this limitation ever go away? I see you have "remove junit from inside testng jar" scheduled. I ask because I don't think people will want to dig into "private" xml files.
Comment 21 Lukas Jungmann 2012-02-06 16:43:54 UTC
(In reply to comment #20)
> If you don't mind, I'd like to reopen this issue until we resolve the basics of
> getting this working in an Ant project.

you mean existing/older ant project with testng tests, right? If so, also update the summary please

> 
> I took my project with existing TestNG tests, removed TestNG from the classpath
> and created a test using the New File wizard. build.xml was modified as you
> mentioned and TestNG was added to the classpath. I then removed the class I
> added using the New File wizard (only my old tests remained) and when I tried
> running with ALT+F6 I got:

the best and fastest would be to attach the project to this issue (or send it to me offline) so I can reproduce the problem on my end...

Have you tried the same with fresh new project with up-to date testng library? If no, can you try that and let me know? (just run IDE with new userdir)

> Instead, I recommend you scan "Test
> Packages" for the @Test annotation. Let me know what you think.

wouldn't it be better if this could be done directly by TestNG itself?

> 
> > (In reply to comment #18)
> > -to get the build script working with arbitrary official testng distro you have
> > to remove 'mixed' from java and testng tasks there
> 
> Will this limitation ever go away? I see you have "remove junit from inside
> testng jar" scheduled. I ask because I don't think people will want to dig into
> "private" xml files.

I hope that it will never go away and it gets included in the official TestNG distro.

Why? Answer is at http://wiki.netbeans.org/TestNG_MixedMode
Comment 22 _ gtzabari 2012-02-06 19:46:23 UTC
(In reply to comment #21)
> the best and fastest would be to attach the project to this issue (or send it
> to me offline) so I can reproduce the problem on my end...
>
> Have you tried the same with fresh new project with up-to date testng library?
> If no, can you try that and let me know? (just run IDE with new userdir)

I fixed the problem by uninstalling the plugin, manually removing changes to build-impl.xml and other files (yuck! this was very error-prone) and reinstalling the plugin.

> > Instead, I recommend you scan "Test
> > Packages" for the @Test annotation. Let me know what you think.
> 
> wouldn't it be better if this could be done directly by TestNG itself?

Agreed.

> > > (In reply to comment #18)
> > > -to get the build script working with arbitrary official testng distro you have
> > > to remove 'mixed' from java and testng tasks there
> > 
> > Will this limitation ever go away? I see you have "remove junit from inside
> > testng jar" scheduled. I ask because I don't think people will want to dig into
> > "private" xml files.
> 
> I hope that it will never go away and it gets included in the official TestNG
> distro.
> 
> Why? Answer is at http://wiki.netbeans.org/TestNG_MixedMode

Did you ask TestNG to integrate this contribution? Is there an official RFE I can vote for?

We can leave this issue closed. I filed a separate bug report for debugging TestNG classes at issue 208105
Comment 23 Lukas Jungmann 2012-02-06 19:53:32 UTC
(In reply to comment #22)
> (In reply to comment #21)
> I fixed the problem by uninstalling the plugin, manually removing changes to
> build-impl.xml and other files (yuck! this was very error-prone) and
> reinstalling the plugin.

yes, this is known to be problematic with plugins from development UC :-(

> > Why? Answer is at http://wiki.netbeans.org/TestNG_MixedMode
> 
> Did you ask TestNG to integrate this contribution? Is there an official RFE I
> can vote for?

I've just announced it @ testng-dev list: https://groups.google.com/group/testng-dev/browse_thread/thread/2f67e742de14fbeb?hl=en so let's see

> 
> We can leave this issue closed. I filed a separate bug report for debugging
> TestNG classes at issue 208105

thanks, that's the best, I'll look at it
Comment 24 _ gtzabari 2012-02-06 20:06:39 UTC
(In reply to comment #23)
> (In reply to comment #22)
> > (In reply to comment #21)
> > I fixed the problem by uninstalling the plugin, manually removing changes to
> > build-impl.xml and other files (yuck! this was very error-prone) and
> > reinstalling the plugin.
> 
> yes, this is known to be problematic with plugins from development UC :-(

What do other plugins do? Shouldn't we have buttons for explicitly installing/uninstalling TestNG integration from a project? You could still implicitly install it if the user uses the File New wizard.
Comment 25 Lukas Jungmann 2012-02-07 13:55:04 UTC
(In reply to comment #24)
> (In reply to comment #23)
> > (In reply to comment #22)
> > > (In reply to comment #21)
> > > I fixed the problem by uninstalling the plugin, manually removing changes to
> > > build-impl.xml and other files (yuck! this was very error-prone) and
> > > reinstalling the plugin.
> > 
> > yes, this is known to be problematic with plugins from development UC :-(
> 
> What do other plugins do? Shouldn't we have buttons for explicitly
> installing/uninstalling TestNG integration from a project? You could still
> implicitly install it if the user uses the File New wizard.

basic trick here to update module's version after each change - I'm about to do that as soon as my proposed patch for testng will be either accepted or rejected - this should inform even released versions of IDE that there's updated version available
Comment 26 _ gtzabari 2012-02-07 15:06:42 UTC
(In reply to comment #25)
> basic trick here to update module's version after each change - I'm about to do
> that as soon as my proposed patch for testng will be either accepted or
> rejected - this should inform even released versions of IDE that there's
> updated version available

That would do it but I'd also like to have the ability to remove plugin integration with my project for good (say I no longer want to use TestNG). Web projects have a "Frameworks" tab in Project Properties. Couldn't we have the same thing for plugins?
Comment 27 Lukas Jungmann 2012-02-07 15:41:48 UTC
You made a good point. 

But I'm thinking about bit different solution for this - w/o any UI - at least at the very beginning[*]. Removing TestNG (or any other test framework) from the project would require removing 3 lines from project.xml... More to come in a wiki ;-)

[*] - Testing specific tab in project properties is maybe already filed somewhere in bugzilla (I'd start searching in junit or j2se project components) - again more about this will be in a wiki