NetBeans and Alternative Test Runners:
Customizing Standard Java Projects

Debug Test File

The Debug Test File override is the most straightforward, so we begin there. The -debug-start-debuggee-test definition calls <debug> which is defined by build-impl.xml as a macro which expands into the full <java> call with internal details. Since we're just supplying a different suite running class, it is straightforward to replace the suite-running class with org.suiterunner.Runner, and supply appropriate parameters to specify the test class. Runner also requires a parameter to specify that we want a default text output reporter that reports failures and aborts of tests, suites, or the run. (Parts specific to SuiteRunner are in bold.)

<target name="-debug-start-debuggee-test"
        depends="init,compile-test,-suiterunner-debug-single"
        if="have.tests" />

<target name="-suiterunner-debug-single">
  <fail unless="test.class">Must select one file in the IDE or set test.class</fail>

  <j3:debug classname="org.suiterunner.Runner"
            classpath="${debug.test.classpath}"
            xmlns:j3="http://www.netbeans.org/ns/j2se-project/3">
    <customize>
      <arg line="-oFBAR"/> <!-- report failures and aborts to System.out -->
      <arg line="-s ${test.class}"/>
    </customize>
  </j3:debug>

</target>

If we add this to a build.xml for a project using SuiteRunner for its test suite, we can right click on a test file and invoke Debug File, and SuiteRunner will be run using the NetBeans debugger, invoking all the public test... methods in the class. If a test fails, SuiteRunner produces output similar the following:

TEST FAILED: SampleTests.testName(): Illegal char in name: Sample_Name
org.suiterunner.TestFailedException: Illegal char in name: Sample_Name
        at org.suiterunner.Suite.verify(Suite.java:853)
        at org.example.SampleTests.testName(SampleTests.java:9)
        ...

NetBeans recognizes the stack trace and produces underlined links to the code. Clicking the second line directs an editor pane to line 9 of SampleTests.java. Sometimes this is enough to correct a bad test.

To use the debugger, set a breakpoint on the failing exception, usually org.suiterunner.TestFailedException. When the test fails the debugger will stop, and you can explore the call stack, look at the values of local variables, and otherwise use the debugger to see what is happening. You may need to set a break pointer earlier in the test method and rerun so that you can step through the methods being tested.

Our first success! If you are in a hurry, this may be all you need to get debugging again. For a little more complete command coverage for SuiteRunner, read on.

Run Test File

Right-clicking on a test file and invoking Run File seems similar to Debug file, but it turns out NetBeans has implemented them differently, making our override of -do-test-run-single a bit more difficult. While Debug File supplied the classname, for Run File the selected file is supplied as a .java filename. In addition, the comments indicate someday there may be multiple selected files rather than a single class, so we'd like it to work with multiple files if possible.

Fortunately, the necessary conversion can be accomplished using Ant <pathconvert>. Each source filename is stripped of its .java suffix, each name is preceded by the -s for a test suite name, and the directory separators are translated to '.' package separators to produce a full classname. (Details are explained in the filepath to classname appendix.)

The Run Test File target can invoke org.suiterunner.Runner using the standard <java> Ant task, supplying the classpath, using org.suiterunner.Runner as the class to run, and providing the converted suite classnames parameter.

<target name="-do-test-run-single" 
        depends="init,compile-test-single,-pre-test-run-single,-suiterunner-run-single"
        if="have.tests" />

<target name="-suiterunner-run-single">
  <fail unless="test.includes">Must select some files in the IDE or set test.includes</fail>

  <!-- convert test.includes "pkg/ClassTests.java" to "-s pkg.ClassTests" -->
  <pathconvert property="suiterunner.classnames.args" pathsep=" " dirsep=".">
    <path><filelist dir="." files="${test.includes}"/></path>
    <mapper type="glob" from="*.java" to="*"/>
    <map from="${basedir}${file.separator}" to="-s "/>
  </pathconvert>

  <java classname="org.suiterunner.Runner"
        classpath="${run.test.classpath}"
        fork="true">
    <arg line="-oFBAR"/> <!-- report failures and aborts to System.out -->
    <arg line="${suiterunner.classnames.args}"/>
  </java>
</target>

If we add this to a build.xml for a project using SuiteRunner for its test suite, we can right click on a test file and invoke Run File, and the file will be run. If a test fails, SuiteRunner produces output similar the following (just like when run in the debugger):

TEST FAILED: SampleTests.testName(): Illegal char in name: Sample_Name
org.suiterunner.TestFailedException: Illegal char in name: Sample_Name
        at org.suiterunner.Suite.verify(Suite.java:853)
        at org.example.SampleTests.testName(SampleTests.java:9)
        ...

NetBeans recognizes the stack trace and produces underlined links to the code (just as it did when the file was run in the debugger). Clicking the second line directs an editor pane to line 9 of SampleTests.java. Sometimes this is enough to correct a bad test.

Test Project

For Test Project, there are two cases depending on whether there is a file that describes which test cases should be run and their order. For SuiteRunner, this file is the recipe.srj file. If there is no such file, then as a default the Ant target should simply run all test files in the test directory.
(Test suite classes using SuiteRunner may have subsuites which are other test suite classes, so a project's SuiteRunner tests may be organized with a root "AllSuites" class rather than a recipe. To run just the "AllSuites" class so suites aren't run more than once, either run the AllSuites class via the Run File command, or create a recipe.srj file for just the AllSuites class.)

If there is a test/recipe.srj file in the NetBeans project, then SuiteRunner can be invoked directly on this file, perhaps as follows:

<target name="-do-test-run" 
        depends="init,compile-test,-post-compile,-pre-test-run,-suiterunner-run-recipe"
        if="have.tests" />

<target name="-suiterunner-run-recipe">
  <echo level="info">Running ${test.src.dir}${file.separator}recipe.srj</echo>

  <java classname="org.suiterunner.Runner"
        classpath="${run.test.classpath}"
        fork="true">
    <arg line="${test.src.dir}${file.separator}recipe.srj"/>
  </java>
</target>

If there is no test/recipe.srj file, then by default the Ant target should run all test classes in the test directory. As a convention, assume that all classes whose names end with ...Test or ...Tests are test classes to include. All test classes extend org.suiterunner.Suite, so also assume class names ending with ...Suite are also test classes to include.

The target needs to collect all class names in the test directory that match these patterns and add them to the command line. As in -do-test-run-single, each name will be preceded by -s on the command line.
(test.src.dir may be a relative path or an absolute path. One of the two <map> clauses below should match the fileset prefixes and replace them with "-s ".)

<target name="-do-test-run" 
        depends="init,compile-test,-post-compile,-pre-test-run,-suiterunner-run-classes"
        if="have.tests" />

<target name="-suiterunner-run-classes">
  <!-- convert "pkg/ClassTests.java" to "-s pkg.ClassTests" -->
  <pathconvert property="suiterunner.classnames.args" pathsep=" " dirsep=".">
    <path>
      <fileset dir="${test.src.dir}" includes="**/*Test.java"/>
      <fileset dir="${test.src.dir}" includes="**/*Tests.java"/>
      <fileset dir="${test.src.dir}" includes="**/*Suite.java"/>
    </path>
    <mapper type="glob" from="*.java" to="*"/>
    <!-- map for relative test.src.dir -->
    <map from="${basedir}${file.separator}${test.src.dir}${file.separator}" to="-s "/>
    <!-- map for absolute test.src.dir -->
    <map from="${test.src.dir}${file.separator}" to="-s "/>
  </pathconvert>

  <echo level="info">Running -oFBAR ${suiterunner.classnames.args}</echo>

  <java classname="org.suiterunner.Runner"
	classpath="${run.test.classpath}"
	fork="true">
    <arg line="-oFBAR"/> <!-- report failures and aborts to System.out -->
    <arg line="${suiterunner.classnames.args}"/>
  </java>
</target>

For running NetBeans on one project with a SuiteRunner test suite, one of the above may be enough to get going. But if some projects have a recipe file and some do not, it would be simpler these two approaches could be combined, so that the target will run the test/recipe.srj file if it exists, otherwise it will run classnames in the test directory. We'll return to that issue after introducing targets for TestNG.

Debug Test File

A basic Debug Test File override is fairly straightforward for TestNG, though there are a few more parameters with details to notice that are specific for TestNG. As with the SuiteRunner definition, the definition for TestNG below uses the NetBeans debug macro. It invokes TestNG with command line parameters. (Parts specific to TestNG are in bold.)

<target name="-debug-start-debuggee-test"
        depends="init,compile-test,-testng-debug-single"
        if="have.tests" />

<target name="-testng-debug-single">
  <fail unless="test.class">Must select one file in the IDE or set test.class</fail>

  <j3:debug xmlns:j3="http://www.netbeans.org/ns/j2se-project/3"
	    classname="org.testng.TestNG" classpath="${debug.test.classpath}">
    <customize>
      <jvmarg line="-enableassertions"/> <!-- TestNG tests often use assert -->
      <jvmarg value="-Dtestng.test.classpath=${build.test.classes.dir}"/>
      <arg line="-d ${build.test.results.dir}"/>
      <arg line="-log 2 -usedefaultlisteners true"/> <!-- show stack traces -->
      <arg line="-sourcedir ${test.src.dir}"/> <!-- for jdk1.4 annotations -->
      <arg line="-suitename ${test.class}"/>
      <arg line="-testname ${test.class}"/>
      <arg line="-testclass ${test.class}"/>
    </customize>
  </j3:debug>
</target>

• -enableassertions is necessary because one standard way to write a TestNG test method is to use an assert, which throws an AssertionError if the test fails. If assertions are not enabled, the test condition of these test methods will not be run.
• testng.test.classpath is an efficiency option that tells TestNG where to look for test classes, rather than the whole classpath.
• -d tells TestNG reporters where to write report files. For NetBeans projects, writing testing reports under the build/test directory is convenient because there they are removed with the build directory when the project is cleaned.
• -log 2 -usedefaultlisteners true tells TestNG to produce the output text that includes stack traces on errors. NetBeans displays source locations in stack traces as clickable links.
• -suitename and -testname are used in the output text and also the report.
• -testclass is the class to test.

After adding this to a build.xml for a project using TestNG for its test suite, a right click on a test file, followed by a click on Debug File, will use the NetBeans debugger to run TestNG on that file. TestNG will run all the test methods annotated with @Test. If a test fails, TestNG can produce output similar the following (where Bad Name: Sample is the text of the assertion message).

FAILED: testName
java.lang.AssertionError: Illegal char in name: Sample_Name
        at org.example.SampleTests.testName(SampleTests.java:9)
... Removed 21 stack frames

NetBeans recognizes the stack trace and produces links to the code. In this case, clicking the line directs an editor pane to line 9 of SampleTests.java. Sometimes this is enough to correct a bad test.

To use the debugger, set a breakpoint on the failing exception, usually java.lang.AssertionError, and run Debug File again. When the test fails the debugger will stop, and you can explore the call stack, look at the values of local variables, and otherwise use the debugger to see what is happening. You may need to set a break pointer earlier in the test method and Debug File again so that you can step through the methods being tested.

Success! If you are in a hurry, this may be all you need to get debugging your TestNG test suite again. If you need to use the TestNG groups feature to run a subset of the test methods in the debugger, see the debug groups appendix. For a little more complete command coverage for TestNG, read on.

Run Test File

As with SuiteRunner, the Run Test File must take its input as a filename, or possibly multiple filenames, in test.includes. The code below converts the filenames to classnames, each preceded by -testclass.
As of TestNG5.4, while the documentation claims that -testclass accepts a comma separated list, that has not been implemented, so each class must be preceded by a -testclass marker.

<pathconvert property="testng.classnames.args" dirsep="." pathsep=" ">
  <path> <filelist dir="." files="${test.includes}"/> </path>
  <mapper type="glob" from="*.java" to="*"/>
  <map from="${basedir}${file.separator}" to="-testclass "/>
</pathconvert>

A similar transformation produces the suitename from the class name (or appending classnames if there are multiple selected classes in the future).

<pathconvert property="testng.suitename" pathsep="_" dirsep=".">
  <path> <filelist dir="." files="${test.includes}"/> </path>
  <mapper type="glob" from="*.java" to="*"/>
  <map from="${basedir}${file.separator}" to=""/>
</pathconvert>

TestNG is invoked with nearly the same parameters as in the Debug Test File case (see description above), but with the transformed classname(s) for -testclass, and for -suitename and -testname.

<java classname="org.testng.TestNG"
      classpath="${run.test.classpath}"
      fork="true" resultproperty="testng.result.code">
  <jvmarg line="-enableassertions"/> <!-- TestNG tests often assert -->
  <jvmarg value="-Dtestng.test.classpath=${build.test.classes.dir}"/>
  <arg line="-d ${build.test.results.dir}"/>
  <arg line="-log 2 -usedefaultlisteners true"/> <!-- show stack trace -->
  <arg line="-sourcedir ${test.src.dir}"/> <!-- for jdk1.4 annotations -->
  <arg line="-suitename ${testng.suitename}"/>
  <arg line="-testname ${testng.suitename}"/>
  <arg line="${testng.classnames.args}"/>
</java>

Clearly, TestNG is invoked via the <java> task rather than <debug>, so it will not be run with the debugger. TestNG also provides a <testng> Ant task. However, when NetBeans runs TestNG using the <testng> task, NetBeans does not create links for source code locations in stack traces in the output, so using the <java> task is preferable.

NetBeans checks the tests.failed property, and ends the run with "BUILD FAILED" if it was set, otherwise with "BUILD SUCCESSFUL". TestNG produces (via System.exit(int)) the result code 0 if there were no test failures, otherwise it produces result code 1. The resultproperty attribute above stores this result in the testng.result.code property. The following code sets tests.failed if the result code is not "0".

<condition property="tests.failed">
  <not> <equals arg1="0" arg2="${testng.result.code}"/> </not>
</condition>

All these parts can now be placed in a -do-test-run-single target in build.xml, overriding the original from nbproject/build-impl.xml.

<target name="-do-test-run-single" 
        depends="init,compile-test-single,-pre-test-run-single,-testng-run-single"
        if="have.tests"/>

<target name="-testng-run-single">
  <fail unless="test.includes">Must select some files in the IDE or set test.includes</fail>
  ... parts described above ...
</target>

With this definition, right clicking on a test file and invoking Run File will run TestNG on tests in the selected file. If a test fails, then it will produce output something like the following, where the underlined stack trace line is clickable and directs the editor to display it, in this case line 9 of SampleTests.java. That may be enough to diagnose a bad test, or to set a breakpoint at the beginning of the test (to step through the problem in the debugger in the next debug session).

FAILED: testName
java.lang.AssertionError: Illegal char in name: Sample_Name
        at org.example.SampleTests.testName(SampleTests.java:9)
... Removed 21 stack frames

At the end of the output, NetBeans reports something like:

BUILD FAILED (total time: 2 seconds)

If no tests failed, then NetBeans reports something like:

BUILD SUCCESSFUL (total time: 2 seconds)

Test Project

For Test Project, there are again two cases depending on whether there is a file that describes which test cases should be run. For TestNG, this file is the testng.xml file. If there is no such file, then as a default the Ant target should run all methods annotated as a @Test in all files in the test directory.

If there is a test/testng.xml file under the NetBeans project directory, then TestNG can be invoked directly on this file, with a target something like below. The parameters are mostly identical to the Debug Test File or Run Test File cases, but instead of supplying -testclass parameters, it provides the testng.xml filepath. The -suitename and -testname parameters are omitted (those names are provided in the testng.xml file).

<target name="-do-test-run" 
        depends="init,compile-test,-post-compile,-pre-test-run,-testng-test-run-xml"
        if="have.tests" />

<target name="-testng-test-run-xml">
  <java classname="org.testng.TestNG"
        classpath="${run.test.classpath}"
        fork="true" resultproperty="testng.result.code">
    <jvmarg line="-enableassertions"/> <!-- TestNG tests often assert -->
    <jvmarg value="-Dtestng.test.classpath=${build.test.classes.dir}"/>
    <arg line="-d ${build.test.results.dir}"/>
    <arg line="-log 2 -usedefaultlisteners true"/> <!-- show stack trace -->
    <arg line="-sourcedir ${test.src.dir}"/> <!-- for jdk1.4 annotations -->
    <arg line="${test.src.dir}${file.separator}testng.xml"/> <!-- after '-' options -->
  </java>

  <condition property="tests.failed">
    <not> <equals arg1="0" arg2="${testng.result.code}"/> </not>
  </condition>
</target>

If there is no test/testng.xml file, then the project needs a target that invokes all the methods annotated with the annotation @Test. TestNG can examine the class files to find those that have @Test annotations, so there is no need to rely on class naming conventions as was done for SuiteRunner, so the pathconvert is simpler.

<target name="-do-test-run" 
        depends="init,compile-test,-post-compile,-pre-test-run,-testng-test-run-classes"
        if="have.tests" />

<target name="-testng-test-run-classes">
  <pathconvert property="testng.classnames.args" pathsep=" ">
    <path>
      <fileset dir="${build.test.classes.dir}" includes="**/*.class"/>
    </path>
    <map from="${basedir}${file.separator}" to="-testclass "/>
  </pathconvert>

  <java classname="org.testng.TestNG"
        classpath="${run.test.classpath}"
        fork="true" resultproperty="testng.result.code">
    <jvmarg line="-enableassertions"/> <!-- TestNG tests often assert -->
    <jvmarg value="-Dtestng.test.classpath=${build.test.classes.dir}"/>
    <arg line="-d ${build.test.results.dir}"/>
    <arg line="-log 2 -usedefaultlisteners true"/> <!-- show stack trace -->
    <arg line="-sourcedir ${test.src.dir}"/> <!-- for jdk1.4 annotations -->
    <arg value="-suitename"/> <arg value="All test classes"/>
    <arg value="-testname"/> <arg value="All test classes"/>
    <arg line="${testng.testclasses}"/>
  </java>

  <condition property="tests.failed">
    <not> <equals arg1="0" arg2="${testng.result.code}"/> </not>
  </condition>
</target>

Invoking TestNG is similar to the Run Test File case, except it uses "All test classes" as the suite and test name.

One of these two targets will work for a project. However, it is more convenient if a target could choose between the two implementations depending on whether a testng.xml file exists in a project. The next section looks at two ways to implement exclusive alternative targets.

Test Project: Depends invocation

Above are two approaches for implementing a Test Project target. One runs tests listed in a file, the other gathers all test classes. These can be invoked by from single Ant super-target by setting a property if the file exists, so the targets that implement each approach can be conditioned on whether or not the property is set.

To set property suiterunner.recipe indicating whether the file test/recipe.srj exists, create a target using the <available> task:

<target name="-check-suiterunner-recipe">
  <available property="suiterunner.recipe"
             file="${test.src.dir}${file.separator}recipe.srj"
             value="${test.src.dir}${file.separator}recipe.srj"/>
</target>

The two conditional targets for the two implementations will use the result of this availability test. Both depend on the -check-suiterunner-recipe target, to make sure they only run after it. The condition in their if or unless property is checked after the depends targets have had a chance to run.

One will run if file recipe.srj exists...

<target name="-suiterunner-run-recipe"
        depends="-check-suiterunner-recipe"
        if="suiterunner.recipe">
  ...
</target>

... and the other will run unless the recipe.srj file exists (in other words, it runs only if file recipe.srj is nonexistent).

<target name="-suiterunner-run-classes"
        depends="-check-suiterunner-recipe"
        unless="suiterunner.recipe">
  ...
</target>

The -do-test-run target makes sure both of these get the chance to run when it does by adding them to its depends list (indirectly via -suiterunner-run for modularity).

<target name="-do-test-run" 
        depends="init,compile-test,-post-compile,-pre-test-run,-suiterunner-run"
        if="have.tests" />

<target name="-suiterunner-run"
        depends="-suiterunner-run-recipe,-suiterunner-run-classes"/>

This is the standard way to encode the dependencies between targets in Ant. Next is an alternative way using <antcall>.

Test Project: <antcall> invocation

The two approaches for implementing a Test Project target described earlier could also be implemented using <antcall>. Rather than using depends to control the ordering requirements, the main target can call the alternative targets directly using <antcall>.

<target name="-suiterunner-run">
  <available property="suiterunner.recipe"
             file="${test.src.dir}${file.separator}recipe.srj"
             value="${test.src.dir}${file.separator}recipe.srj" />
  <!-- just one of these runs, depending on existence of test/recipe.srj -->
  <antcall target="-suiterunner-test-run-recipe"/>
  <antcall target="-suiterunner-test-run-classes"/>
</target>

The two conditional targets for the two implementations can depend on the result of this availability test. One will run if the recipe.srj file does exist...

<target name="-suiterunner-run-recipe"
        if="suiterunner.recipe">
  ...
</target>

... and the other will run unless the recipe.srj file exists (in other words, it runs only if the recipe.srj file does not exist).

<target name="-suiterunner-test-run-classes"
        unless="suiterunner.recipe">
  ...
</target>

Unlike the 'depends' case, in this case there are no depends attributes; the order is hard-coded in the target above. Also, notice the <available> check must be written inline in the -suiterunner-test-run target.

Trade-offs

The <antcall> version seems shorter, and for programmers used to writing sequential procedures, more straightforward. However, <antcall> is not simply a subroutine call. In addition to being invoked, the target of an <antcall> also receives a private set of all property bindings, so it can modify them as needed for the sub-target without interfering in the caller's state. This means that modifications made by the sub-target are not visible to the caller. That is why in the <antcall> version, the <available> check is made inline, not in a sub-target that was target of an <antcall>.

For NetBeans, the target running the tests can set the property tests.failed to indicate to NetBeans whether any tests failed. If some tests failed, then NetBeans will report "BUILD FAILED", and if no tests failed, NetBeans will report "BUILD SUCCESSFUL". However, if this target is invoked via an <antcall>, then any changes will be discarded when it returns, so the property NetBeans sees will not be set and NetBeans will always report "BUILD SUCCESSFUL" even if some test failed.

(Unfortunately, currently that is not the end of the story. The NetBeans CodeCoverage module uses the Emma code coverage package to help find code blocks that have been missed by a test suite. It affects a test run as follows: After compilation it instruments the non-test class files, so that when the tests are running they record which blocks were called by the tests. After the test run completes, it scans the recorded data and colors the non-test source code to indicate which lines ran sometime during the tests. The problem is that with NetBeans 5.5, codecoverage 2.2.1, and the depends version of the Test Project target, the instrumentation doesn't occur until after the tests have run. I have not found the reason for this difference, but running with the <antcall> version works around the problem. Maybe creating a copy of all the properties somehow triggers the instrumentation target.)