Bug 45696 - Slow classpath building with referenced paths.
Summary: Slow classpath building with referenced paths.
Status: NEW
Alias: None
Product: Ant
Classification: Unclassified
Component: Core (show other bugs)
Version: 1.7.1
Hardware: PC Windows XP
: P2 enhancement with 1 vote (vote)
Target Milestone: ---
Assignee: Ant Notifications List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-08-27 09:12 UTC by Constantine Plotnikov
Modified: 2009-08-27 07:13 UTC (History)
0 users



Attachments
The archive with sources and build scripts that reproduce the bug. (29.62 KB, application/octet-stream)
2008-08-27 09:12 UTC, Constantine Plotnikov
Details
The archive with sources and build scripts that reproduce the bug with additional layer of projects. (35.93 KB, application/octet-stream)
2008-08-28 06:37 UTC, Constantine Plotnikov
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Constantine Plotnikov 2008-08-27 09:12:06 UTC
Created attachment 22487 [details]
The archive with sources and build scripts that reproduce the bug.

When generating ants scripts, we have discovered that if classpath for java application references other paths using <path refid="<path name>"/> a significant slow down happens for big projects with complex dependency structure for launched java applications and tests. We had 15 minutes for classpath building during execution of test tasks.

The stack probed from profiler looks like the following:

java.lang.Class.isAssignableFrom(Class)
org.apache.tools.ant.types.DataType.getCheckedRef(Class, String, Project)
org.apache.tools.ant.types.DataType.getCheckedRef(Project)
org.apache.tools.ant.types.DataType.getCheckedRef()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.Path.isFilesystemOnly()
org.apache.tools.ant.types.resources.BaseResourceCollectionContainer.isFilesystemOnly()
org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection)
org.apache.tools.ant.types.Path.iterator()
org.apache.tools.ant.types.Path.iterator()
org.apache.tools.ant.types.resources.Union.getCollection(boolean)
org.apache.tools.ant.types.resources.Union.list()
org.apache.tools.ant.types.Path.list()
org.apache.tools.ant.types.Path.addExisting(Path, boolean)
org.apache.tools.ant.types.Path.addExisting(Path)
org.apache.tools.ant.types.Path.concatSpecialPath(String, Path)
org.apache.tools.ant.types.Path.concatSystemClasspath(String)
org.apache.tools.ant.types.CommandlineJava.haveClasspath()
org.apache.tools.ant.types.CommandlineJava.addCommandsToList(ListIterator)
org.apache.tools.ant.types.CommandlineJava.getCommandline()
org.testng.TestNGAntTask.executeAsForked(CommandlineJava, ExecuteWatchdog)
org.testng.TestNGAntTask.execute()

The method org.apache.tools.ant.types.Path.assertFilesystemOnly(ResourceCollection) looks like the primary CPU eater due to recursive invocation.

The attached file contains project that allows to reproduce the problem.
There are two ant scripts:

The script build-inlined.xml demonstrates situation with inlined classpaths. The run task is executed very fast (ant reports 0 seconds).
 
The script build-ref.xml demonstrates situation with referenced classpaths. The run task is executed for about (3 seconds) and the slowdown is due to classpath handling in ant application itself.

The only difference between these tasks is the way classpaths for java task are specified in the "run" target.
Comment 1 Matt Benson 2008-08-27 09:18:27 UTC
Slow but correct behavior is undesirable but I'm not sure if it qualifies as a bug.
Comment 2 Matt Benson 2008-08-27 09:32:49 UTC
Also, it looks entirely like your problem is not the use of one ref, but the fact that in your "bug" files your classpath is composed of sixteen other paths. Finally, your examples seem to suffer from their own smallness:  the time it takes me to run either (against svn trunk) is 12 seconds.
Comment 3 Constantine Plotnikov 2008-08-27 10:43:42 UTC
Try to run in situation when sources are already compiled to see the difference. The problem is with "run" target. Compilation uses inlined classpath.

In the real complex project we got 15 minutes. So there is likely non-polynomial dependency on the size of classpath. 16 paths are actually very small amount (especially considering that they are one element). 
Comment 4 Constantine Plotnikov 2008-08-27 10:49:47 UTC
> small amount (especially considering that they are one element). 
Ops. I meant "one new element each".

Comment 5 Constantine Plotnikov 2008-08-28 06:37:20 UTC
Created attachment 22494 [details]
The archive with sources and build scripts that reproduce the bug with additional layer of projects.

I have added an additional layer of modules (e1-e2) to the project and created new build scripts. Now run target for ref case takes 30 seconds on my computer (vs 0 for inline case).
Comment 6 Stefan Bodewig 2009-08-27 07:13:13 UTC
svn revision 808421 adds a cache attribute on <path>.

If I enable it on

  <path id="antbug.runtime.module.classpath" cache="true">

my build time goes from 32 seconds to 13 seconds using your example.  If I set it on all paths with "runtime" in their name it goes down to 7 seconds.

This is still worse than the 2 seconds I get without refs but I don't really see how to improve the situation any further since most of the time is spent making sure we don't have a cycle in the graph of connected references.

Given the improvement now, I'm going to lower the importance, but not closing the issue since others may find more tweaks we could apply.