Bug 55697 - JUnit tests skipped by haltonerror or haltonfailure are missing from skipped test reporting
Summary: JUnit tests skipped by haltonerror or haltonfailure are missing from skipped ...
Status: NEW
Alias: None
Product: Ant
Classification: Unclassified
Component: Optional Tasks (show other bugs)
Version: nightly
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Ant Notifications List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-10-23 23:36 UTC by Greg Briggs
Modified: 2013-12-03 05:55 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Greg Briggs 2013-10-23 23:36:59 UTC
The ability to report skipped tests was recently added to the Ant JUnit reporting (in March 2013 / bug 54670). When the "haltonerror" or "haltonfailure" directives are used in the junit task, and the build does halt before running all the tests, then the resulting skipped tests should be reported via this mechanism as well. (Whether you consider this a bug or a feature, it would still be good.)

Software versions used to repro:
 Apache Ant(TM) from SVN, version 1.9.3alpha retrieved and compiled on October 21 2013
 Apache Ant(TM) version 1.9.2 also has this issue
 JUnit 4.11 (with Hamcrest-core 1.3)
 Java version 1.7.0

How to reproduce:
 Create the following files and then run "ant" in the directory. Note that the test "TestFail.test3" was skipped, but there is no XML report of this getting skipped. The build directory should contain a report stating that a test was skipped.

== build.xml ==
<project name="Repro" default="tests" basedir=".">

  <target name="compile">
     <mkdir dir="build"/>
     <javac srcdir="src" destdir="build">
        <classpath><path path="../hamcrest-core-1.3.jar:../junit-4.11.jar"/></classpath>
     </javac>
     <jar jarfile="Repro.jar" basedir="build"/>
  </target>
  
  <target name="tests" depends="compile">
     <junit 
        haltonerror="true" haltonfailure="true"
        outputtoformatters="true" showoutput="true">
        <formatter type="xml" usefile="true"/>
        <test name="com.amazon.TestFailFail" todir="build"/>
        <test name="com.amazon.TestFail" todir="build"/>
        <classpath><path path="../hamcrest-core-1.3.jar:../junit-4.11.jar:Repro.jar"/></classpath>
     </junit>
  </target>

  <target name="clean">
     <delete dir="build"/>
     <delete file="Repro.jar"/>
  </target>

</project>

== src/com/amazon/TestFailFail.java ==
package com.amazon;
import static org.junit.Assert.fail;
import org.junit.Test;

public class TestFailFail
{
    @Test public void test1()
    { fail("I am test1 fail."); }

    @Test public void test2()
    { fail("I am test2 fail."); }
}

== src/com/amazon/TestFail.java ==
package com.amazon;
import static org.junit.Assert.fail;
import org.junit.Test;

public class TestFail
{
    @Test public void test3()
    { fail("I am test3 fail."); }
}
Comment 1 Michael Clarke 2013-12-01 10:58:35 UTC
I'm not sure there is a suitable fix for this.

If one of the classes that hasn't been executed was to return a suite (therefore providing more tests to be run), be invoked by a runner that identifies methods differently (different annotations or by alternative naming conventions), use parameterised tests, or anything else that alters the number of tests beyond what can be counted statically with the @Test annotation then we're going to fail to count these tests properly.

We can count methods annotated with @Test (JUnit 4) or named test* (JUnit 3) in un-executed classes, but this may lead to an inconsistent count when compared to a successfully run suite. The ignored count was developed for tests that the user had specifically ignored whilst tests were being executed, not that Ant hadn't tried to execute, and I think trying to change it is just going to cause further issues.

I therefore propose closing this as with a "won't fix" status.
Comment 2 Stefan Bodewig 2013-12-03 05:19:37 UTC
The alternative is to document the count is a "best guess".
Comment 3 Greg Briggs 2013-12-03 05:55:57 UTC
I would propose that the existing JUnit functionality be used to enumerate the tests. We should not be duplicating logic about test case discovery in Ant. The JUnit 4 Runner interface includes a method "testCount()" which returns the number of tests to be run. Ant could instantiate the Runners to see how many test cases exist, and simply not call run() as it usually would have. This covers the Parameterized test use case as well.