This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 116032 - Ability to distribute reusable test components of modules
Summary: Ability to distribute reusable test components of modules
Status: RESOLVED WONTFIX
Alias: None
Product: apisupport
Classification: Unclassified
Component: Harness (show other bugs)
Version: 6.x
Hardware: All All
: P3 blocker with 1 vote (vote)
Assignee: Jesse Glick
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-09-19 16:25 UTC by tomwheeler
Modified: 2010-05-05 15:24 UTC (History)
2 users (show)

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description tomwheeler 2007-09-19 16:25:23 UTC
Premise 1: Good applications require automated testing to ensure they work as designed
Premise 2: A complete setup should not include just unit tests, but also UI, performance and other types of tests
Premise 3: The platform should make it easy to do the right thing

Therefore: The platform should make it as easy as possible to support all types of automated testing, not just unit tests.

Currently, the NetBeans platform has built-in support only for JUnit tests.  However, various NetBeans projects (Jemmy,
Jelly, Insane, etc.) have been developed to support other types of testing needed to fully verify that the applications
work as intended.  I'd guess that more than half of the members of the dev@openide list have even heard of XTest and
probably only 10% or less are successfully using it.  It would be far easier for developers and more likely to get used
if it (or some equivalent facility for running these types of tests) was there by default (i.e. in the harness and not
coupled to the IDE).
Comment 1 Jesse Glick 2007-09-19 19:02:03 UTC
There is in fact an XTest support module available on AU you can use, as well as some other modules to help set up
tests. I know little about the status or details. I think Zajo would know more or could pass it to the right person.
Comment 2 tomwheeler 2007-09-19 19:36:41 UTC
Thanks, Jesse.  I appreciate your comments, though my goal is perhaps a bit different.  I am not looking for any modules
to install into the IDE (although that would be perhaps a nice complement to what I am suggesting).

Currently you can create a new platform application and it will have a build harness that supports running JUnit.  You
don't have to do anything extra to enable this; the support is "just there."  What I am saying in this issue is that the
harness for platform-based applications ought to support testing beyond simple JUnit tests, specifically Jemmy, Jelly,
Insane as well as creating coverage reports.  

All of that is available separately now in XTest, but should be somehow integrated into (or at least their targets
readily available from) the build harness created for a new platform app so that you can run Jemmy, NBJunit and other
types of tests "out of the box" using just Ant as you can now with JUnit.
Comment 3 Jesse Glick 2007-09-19 19:45:25 UTC
All of these frameworks introduce substantial added complexity, size, and failure points; so there are no plans to make
any changes to the contents of the default harness, though add-on integration modules can provide good support for
people who want to use these frameworks.
Comment 4 pzajac 2007-10-24 17:30:33 UTC
Xtest, jemmy, and jellytools are available on AUC, it's right. Xtest is proprietary but only one harness supported in
NetBeans. All the tools are handy for nbm.org. It's not trivial to use it for external modules. 

I vote to add at least nbjunit.jar library o default harness. Xtest is overcomplicated harness but the only one which
allows to run tets in IDE. I think that NB platform users prefer to create tests to be run in running platform than in
unit mode. Unit tests are for experienced developers. For example: who know, how to set up default lookup in module
suite? Mock servervices are not available...    

IMHO the best guy for integration xtest to harness is Jiri Skrivanek. Jiri Skrivanek maintains Xtest for nb.org. I am
not afraid that it takes more resources to maintain it for external modules. Xtest, jemmy, jellytools, nbjunit are used
many developers and testers of NetBeans team. We can expect that it is well tested. 

  
Comment 5 Jesse Glick 2007-10-24 18:23:27 UTC
I am not in favor of adding just nbjunit.jar to the harness; in fact I am in favor of removing its special-case status.
It should be treated as a test dependency of the module that you have to declare like any other. MockServices is not
even necessary any more if you have MockLookup, and there are more and more useful test facilities in other modules.

Currently we have no way of packaging test libraries for external use. They are not even separated from the actual tests
in various modules. This situation needs to be resolved somehow. For example, org.openide.util.test.* and
org.openide.filesystems.test.TestFileUtils are generally useful and could be distributed as standalone test CP
additions, if made into separate projects (?).

Running unit tests in "standalone" mode is always preferable when it is possible. There are really several tiers, moving
from less complicated to more complicated (and unreliable):

1. Plain unit tests which just load the classes they need from the classpath. As of NB 6.0, META-INF/services lookup and
XML layers will be available automatically.

2. Unit tests which call Lookup.getDefault().lookupAll(ModuleInfo.class) to load the NB module system. Some core JARs
must be on CP; any modules found on CP will be loaded. Will add Services/ folder to default lookup, ModuleInstall's will
be run.

3. IDE-mode tests which currently require XTest but show no GUI. (For example, ValidateLayerConsistencyTest.) Only value
over #2 that I can see is that all application modules are loaded without the need to explicitly enumerate test
dependencies - thus useful for app-wide validation-type tests which check for the existence of problems in arbitrary
other modules.

4. IDE-mode functional tests which show the main window and run Jemmy/Jelly-based GUI tests.

IMHO we should move away from using XTest to run IDE-mode tests and develop a simpler, lightweight alternative for
running the body of a unit test in a live IDE (or whatever other application). You would need to make the test
platform's installation location available to the unit test, perhaps as a system property, and somehow instruct JUnit to
run its body in a special way (perhaps by extending a special TestCase subclass). Several possibilities come to mind:

a. The TestSuite (i.e. one Java source) starts an instance of the app with a fresh user directory, runs the test cases,
shuts down the app. This is simplest. It might be slower if you have a lot of different suites (classes) in one module,
though on the other hand you get isolation between suites which is always desirable.

b. While connecting to a running app, or starting one on demand, is certainly possible with NB (use CLIHandler!), there
is no obvious way to know when to shut down the app because no test suite is going to know when it is the last to be run
in a given session. One strategy would be to simply wait for e.g. 60 seconds after a test suite has disconnected, and if
there are no further connections, shut down. This might however lead to file locking problems on Windows, or random
errors on other OSs, if the IDE tries to write out any settings (e.g. window configuration) after a test suite has been
run and its working area deleted. You could try to solve this by using a userdir in /tmp, rather than inside a project
build dir, and trying to get the IDE to delete as much as possible of the userdir as the last thing it does before exiting.

c. Use approach (a) with the modification that the user creates a TestSuite subclass for the module which includes a
number of classes all to be run in the same app VM.

I think taking any of these approaches, and distributing the resulting library (just a JAR to add to your test CP),
would be easier than trying to hack XTest any further.
Comment 6 pzajac 2007-10-25 09:45:09 UTC
nbjunit.jar - module
There are few problems. nbjunit.jar is on testing classpath at default now. If you create module from nbjunit.jar you
did uncompatible change. It's simply fixed.

Packaging test libraries
Binary test distribution was developed. It's mean that all tests of a module and type (unit or qa-functional ) are
packed to jar file. The jar file is placed to folder:
${test.dist}/${test.type}/${cluster}/${code.name.base}/tests.jar

We just only need extra specification - that a package of a tests.jar contains test API. We can use for it a properties
file with items:
unit/platform7/org-openide-util/tests.jar.public_packages=org.openide.utils.test,org.openide.utils.test2

4. IDE-mode functional tests which show the main window and run Jemmy/Jelly-based GUI tests.

Removing dependency on xtest:
I fully agree with removing xtest hacking. Junit unit harness has problems with killing tests. I'm afraid that it's too
complicated for nb.org modules to do it. 

>c. Use approach (a) with the modification that the user creates a TestSuite subclass for the module which includes a
> number of classes all to be run in the same app VM.



 How can you distinguish between IDE and junit test? Is not better to create rule:
unit tests - unit test-type
ide tests - qa-functional

It saves a lot of work with migration. 

Cli handler - we were already together with jtulach thinking about it in past. Extra test module can be created and
deployed by CLI.

> You could try to solve this by using a userdir in /tmp
With tmp folder are many complications in multiuser environment :(. I am rather for extra workdir or build folder.

Comment 7 Jesse Glick 2008-06-23 16:53:36 UTC
Changing description of issue since the main points have already been addressed in the simpletests branch.
Comment 8 Jesse Glick 2008-12-10 00:32:01 UTC
Probably WONTFIX. I regret that test-to-test dependencies were ever introduced to begin with; has caused a lot of headaches.
Comment 9 Jesse Glick 2010-05-04 23:26:10 UTC
You can distribute test components as regular modules and make test deps on them.
Comment 10 tomwheeler 2010-05-05 15:24:23 UTC
Agreed.  The platform build harness now supports (or at least doesn't preclude me from adding support for) all the testing I typically need to do.