Bug 43969 - JUnit4 tests marked @Ignore do not appear in XML output
JUnit4 tests marked @Ignore do not appear in XML output
Product: Ant
Classification: Unclassified
Component: Optional Tasks
Other other
: P2 normal with 24 votes (vote)
: 1.9.0
Assigned To: Ant Notifications List
: PatchAvailable
Depends on:
  Show dependency tree
Reported: 2007-11-26 15:54 UTC by Dan Fabulich
Modified: 2014-02-17 13:49 UTC (History)
8 users (show)

sample project (141.81 KB, application/x-zip-compressed)
2007-11-26 15:54 UTC, Dan Fabulich
resulting test output (6.41 KB, text/plain)
2007-11-26 15:55 UTC, Dan Fabulich
Proposed Skipped Test Reporting Changes (42.44 KB, patch)
2013-03-03 22:28 UTC, Michael Clarke
Details | Diff
Updated Patch (38.82 KB, patch)
2013-03-04 20:42 UTC, Michael Clarke
Details | Diff
Example Patched XML Output (7.59 KB, text/plain)
2013-03-04 20:52 UTC, Michael Clarke
Updated patch (40.97 KB, patch)
2013-03-04 21:51 UTC, Michael Clarke
Details | Diff
Updated Patch +test -duplication (43.23 KB, patch)
2013-03-04 22:49 UTC, Michael Clarke
Details | Diff
Ant junit xsl style sheets modified to show skipped tests (11.05 KB, application/octet-stream)
2013-03-21 10:30 UTC, Gilles Gaillard

Note You need to log in before you can comment on or make changes to this bug.
Description Dan Fabulich 2007-11-26 15:54:08 UTC
Run a JUnit 4 test marked with the @Ignore annotation.  The test will not appear
at all in the XML output.  (I'll attach a sample project.)
Comment 1 Dan Fabulich 2007-11-26 15:54:46 UTC
Created attachment 21190 [details]
sample project
Comment 2 Dan Fabulich 2007-11-26 15:55:17 UTC
Created attachment 21191 [details]
resulting test output
Comment 3 Dan Fabulich 2007-11-26 16:18:48 UTC
I file this bug mostly because, as far as I know, the Ant team "owns" the format
of JUnit XML that's consumed by various other tools (including Maven, Eclipse,
CruiseControl, etc.)  I don't know what the "expected" XML should look like
(though it's pretty obvious that what we've currently got is wrong).

Maybe something like this?

<testsuite total="1" ignored="1">
  <testcase name="testIgnorable>
    <ignore message="reason"/>

TestNG does this:

<testsuite total="1" skipped="1">
  <testcase name="testIgnorable>
    <skipped message="reason"/>
Comment 4 Steve Loughran 2007-11-27 02:33:01 UTC

as discussed on testng-user, this is because the current xml+report is junit3.x
era. I am willing to extend it to support junit4, testng, antunit,
smartfrog-xunit tests, etc, as long as we can do this without breaking :
CruiseControl, Hudson, et al. 

Let's start by creating a wiki entry and listing 
 -tools/products that generate the report
 -tools/products that process the XML it generates
 -features wanted 
 -test plan

Can I also point you at

where I show a prototype XHTML format that can be streamed out and is readable
before the tests have finished. Moving to something like that would break
everything, so I'm not going to propose it. But I would like to enable better
streaming, by pushing all new summary information out as elements and not
attributes on root nodes. 

I'd like something such as

<testsuite total="1">
    <description>Tests of OS-specific things</description>
     <rdf xmlns:rdf="http://w3c.org/rdf">
      <!-- random triples here -->

 <testcase name="testIgnorable">
     <description>A test that only works on Win98</description>
     wrong OS


1. new summary data goes in at the end. If you want to handle skipping, embrace
the new format

2. extra stuff goes into a metadata corner. This is all static with every test.
I'd like everything needed to create interesting Atom feeds.
Comment 5 Dan Fabulich 2007-11-27 10:52:21 UTC
Steve has created a wiki article here:

Comment 6 Jesse Glick 2007-12-03 13:46:52 UTC
May not be straightforward to fix in the current <junit> task, which generally
assumes JUnit 3.x and only handles 4.x tests using the compatibility wrapper.
Real support for 4.x should I think be a separate task.
Comment 7 Ankur 2008-12-01 13:22:35 UTC
We are planning to migrate all our test cases to Junit4. Is this issue resolved yet? If not, are there any timelines on when will this feature be available. I would appreciate if I could get an update on this or if there is a workaround.
Comment 8 Stefan Bodewig 2008-12-01 21:22:07 UTC
I'm not aware of anybody working on a JUnit4 specific Ant task, for most people our JUnit3 task seems to work well enough.

So no, it is not fixed and can't be fixed easily since Ant's approach to JUnit4 makes the @ignore-d tests invisible to Ant.  And it doesn't seem to be high on anybody's priority list either.
Comment 9 Gilles Gaillard 2010-01-13 03:14:11 UTC
It's a pity that it's not high a in the priority, although the wiki proposal mentioned 1.8 as a possible target.

Also, it seems that this tickets is not simply to support the @Ignore but to allow full support to junit-4 in ant. 
Hence, suggesting that in the updated <junit> task, a 'run-with' element would be welcome, allowing to force the junit-4 @RunWith in tests.
Comment 10 Mario Frasca 2011-04-21 08:18:10 UTC
I have it working in R (svUnit, integrates nicely with Jenkins).
This unit test suite manages to report a test as ignored by adding a further totally empty `skipped` element in the `testcase` element.

Comment 11 Harry Soehalim 2012-08-14 22:11:48 UTC
Hi There - is there any update on this bug? Thanks, Harry.
Comment 12 Michael Clarke 2013-03-03 22:28:22 UTC
Created attachment 30011 [details]
Proposed Skipped Test Reporting Changes
Comment 13 Michael Clarke 2013-03-03 22:38:43 UTC
I've added a patch with changes to report Ignored test (those marked with the @Ignore annotation) and skipped tests (those where Assume tests failed). The reporters bundled with the junit task have been modified to report these tests as skipped, and an additional interfaces has been added to allow users to add these events to their existing TaskListeners.

The default XML report of changes contains the follow additional details with my patch:
1. An additional attribute on the root testsuite element named 'skipped' that contains the number of ignored and skipped tests
2. The test attribute on the root testsuite element now counts all tests (both run and ignored), not just the ones that have been run
3. All tests are listed as testcase elements, not just the ones that haven't been ignored
4. Any test that has been ignored or skipped has a child 'skipped' element that, if a message was given on the @Ignore annotation or a message given for an Assume failing, includes a message attribute containing that message

This bring the XML document in-line with the XMl reports from other runners (e.g. Maven Surefire) and other test suites (e.g. TestNG)

Patch attached as 'Proposed Skipped Test Reporting Changes'. Any comments, ideas, suggestions or issues are welcome.
Comment 14 Michael Clarke 2013-03-04 20:42:29 UTC
Created attachment 30012 [details]
Updated Patch

Updated patch:

1. Separate JUnit4 code out from formatters so users can continue to run using JUnit3
2. Support retrieving @Ignore messages from JUnit4 versions prior to version 4.4

This new patch has been tested against both JUnit3 and JUnit4 with no issues.
Comment 15 Michael Clarke 2013-03-04 20:52:20 UTC
Created attachment 30013 [details]
Example Patched XML Output

Example XML output with explicitly ignored test (using @Ignore) and skipped tests (failed Assume), as well as errors, failures and successes.
Comment 16 Michael Clarke 2013-03-04 21:51:20 UTC
Created attachment 30014 [details]
Updated patch

Patch with test case for checking XML format re-instated
Comment 17 Michael Clarke 2013-03-04 22:49:23 UTC
Created attachment 30015 [details]
Updated Patch +test -duplication

Patch with test case re-instated and duplicate code removed.

The previous patch introduced a CustomJUnit4TestAdapterCache which provided the notifier for Ignored and Skipped events in JUnit4. This new class mostly duplicated the existing private 'Cache' class within JUnit4TestMethodAdapter, although added the methods needed for Ignored and Skipped events. This latest patch removes the private class and changes references to it to the new public class so as to prevent code duplication.
Comment 18 Antoine Levy-Lambert 2013-03-05 05:43:26 UTC
Fixed. Thanks Michael for the big contribution.
Comment 19 Gilles Gaillard 2013-03-21 10:30:09 UTC
Created attachment 30092 [details]
Ant junit xsl style sheets modified to show skipped tests

Attached modified junit-frames.xsl and junit-noframes.xsl to show skipped count.

In 'junitreport' task, use the 'styledir' attribute of the 'report' element to specify the directory where to find the style sheets.
Comment 20 Michael Clarke 2013-03-21 10:59:01 UTC
Gilles: I've already raised a further issue for this with an associated patch - Bug 54670. I've not checked your patch but assume it's similar to what I've done, if not then feel free to attach it to the new issue.