Bug 45499 - DirectoryScanner infinitely recurses on symlinks to parent directories
Summary: DirectoryScanner infinitely recurses on symlinks to parent directories
Status: RESOLVED FIXED
Alias: None
Product: Ant
Classification: Unclassified
Component: Core (show other bugs)
Version: 1.7.1
Hardware: Macintosh Mac OS X 10.4
: P2 critical (vote)
Target Milestone: 1.8.0
Assignee: Ant Notifications List
URL:
Keywords: PatchAvailable
Depends on:
Blocks:
 
Reported: 2008-07-29 19:04 UTC by Tim Whittington
Modified: 2008-10-28 05:01 UTC (History)
1 user (show)



Attachments
Patch to DirectoryScanner to prevent recursion on symlinks to ancestor dirs (1.06 KB, patch)
2008-07-29 19:04 UTC, Tim Whittington
Details | Diff
logs from compile task (76.33 KB, text/plain)
2008-10-16 13:47 UTC, Alvaro
Details
DWR2 build logs (279.29 KB, text/plain)
2008-10-17 15:05 UTC, Alvaro
Details
build.properties and build.xml compressed (3.05 KB, application/zip)
2008-10-17 15:07 UTC, Alvaro
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Whittington 2008-07-29 19:04:24 UTC
Created attachment 22327 [details]
Patch to DirectoryScanner to prevent recursion on symlinks to ancestor dirs

I'm using Ant 1.7.1 on Mac OS X 10.5.4 with Java SE 6 (1.6.0_05).

When compiling some fairly basic Java source using the javac task I get consistent OutOfMemoryErrors. Some debugging reveals that this is due to an infinite recursion of directories in DirectoryScanner.

The Java framework installed on the Mac has a Home directory that emulates the layout of a JRE on *nix/Windows, so apps (like Ant) that like to base things off $JAVA_HOME can find things.

In Java 6, this Home directory has two symlinks to parent directories, namely:
bundle -> ../
shared_bundle -> ../../../
(This is a default install of the latest Java update for OS X 10.5 - I haven't modified it in any way, so this issue will probably be an issue for all OS X users).

During the javac execution, DirectoryScanner seems to scan the $JAVA_HOME directory (I haven't tried to ascertain why), resulting in an infinite recursion on bundle -> Home -> bundle -> Home etc. etc., eventually ending in an OOME.

I'm attaching a patch that resolves listed child directories to canonical paths and checks if they're leaders of the currrent director, which fixes this issue.
Comment 1 Stefan Bodewig 2008-09-03 07:50:04 UTC
I'm not convinced that ignoring the symlink completely is the right thing to do, we probably should follow it but make sure we don't scan directories we've already seen again.
Comment 2 Matt Benson 2008-09-03 07:52:36 UTC
Generally a lot of the internals of DirectoryScanner are private, so we can likely change signatures to pass around an e.g. Stack if needed.
Comment 3 Matt Benson 2008-09-03 08:28:39 UTC
looks like that won't work after all (protected scandir), but DS is written in a stateful way, so something could be done.
Comment 4 Tim Whittington 2008-09-03 19:04:48 UTC
(In reply to comment #1)
> I'm not convinced that ignoring the symlink completely is the right thing to
> do, we probably should follow it but make sure we don't scan directories we've
> already seen again.
> 

It looks like DirectoryScanner already avoids scanning directories it's already encountered (via hasBeenScanned(String)). I haven't verified this actually works as intended though, and it only seems to fire on fast scans.

I'd struggle to imagine a real use case for a symlink to a parent directory that I'd actually want to recurse through, but it'd be sensible if you're looking for a bullet-proof general solution.
Comment 5 Stefan Bodewig 2008-09-04 00:50:38 UTC
Matt: yes, I think it is doable, even without too many or too big changes, but we need to think through some scenarios first to decide what to do.  I'll take that to the dev list later (after coding up some AntUnit tests as use cases).

Tim: hasBeenScanned only receives the relative path which will be different each time the directory is encountered again.

I was thinking about a case where we have a structure like this

A/
|---> B/
      |-----> C.txt
      D/
      |-----> E (symlink pointing to A)

Let's say we set basedir to D and scan with an include pattern of E/B/C.txt - should it match the file or should we skip going upwards (we wouldn't rescan D since it isn't matched by an include pattern)?  I haven't checked, but we may even get the file in fast mode and an infinite loop in slow mode (when the client asks for the not-included files/dirs).

The good thing is that we don't need to think about backwards compatibility for the cases where Ant could run into an infinite loop 8-)

Like I said, I'll move the discussion over to the dev list.
Comment 6 Stefan Bodewig 2008-09-04 07:30:05 UTC
dev thread started http://marc.info/?l=ant-dev&m=122053158307830&w=2
Comment 7 Stefan Bodewig 2008-09-10 08:44:32 UTC
the infinite loop should be gone with svn revision 693870.

<fileset> now has a new attribute currently named maxLevelsOfsymlinks that currently defaults to 1.  It controls how often a given symlink will be followed.

Not yet documented (that's why I don't close the report) since attribute name
and default value are up for discussion.
Comment 8 Stefan Bodewig 2008-09-16 01:41:56 UTC
Tim, would it be possible for you to download the latest source snapshot from http://svn.apache.org/snapshots/ant/ , build it and verify that it works for you?
Comment 9 Tim Whittington 2008-09-22 03:17:53 UTC
I downloaded and built the ant_20080922041906.tar.gz snapshot, and ran it against my build, and the compile now appears to complete successfully.

I'd really like not to be spammed as much for my one offending symlink though (there's a bit of exponential explosion going on with the potential paths it gets to skip)

In a probably unrelated issue, my build is now failing with the new snapshot (it appears to have broken a fairly innocuous looking AntX foreach, complaining the loop property isn't set).

Log excerpt follows.

    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Home -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.1/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6.0/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6.0/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/CurrentJDK/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/CurrentJDK/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Home -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.1/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.2/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5.0/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6.0/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6.0/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6.0/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/CurrentJDK/Home/bundle/Home/bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/CurrentJDK/Home/bundle/Home/shared_bundle -- too many levels of symbolic links.
    [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/CurrentJDK/Home/shared_bundle -- too many levels of symbolic links.
[antx:callforeach] antx:callforeach unable to run all steps; step [copy.inline.resources] threw failure.
[antx:callforeach] Unable to complete steps when loop cursor at [/Users/archie/Projects/OrionPlatform/Tools/com.orchestral.common.hibernate3/src/main/java].
Comment 10 Stefan Bodewig 2008-09-25 05:55:46 UTC
more recent versions will allow the same symlink to be traversed more often (to reduce the performance and memory impact of the tests), so your build will consume a bit more memory.

We might lower the log priority, by writing the message to System.out, but it would probably remain visible anyway.  I'm afraid there isn't much we can do.
Comment 11 Alvaro 2008-10-15 08:14:08 UTC
I have checkout the trunk from SVN (october 14) compile it and tried to compile a project and the problem is present yet. I have the same logs as in the  Comment #9, with the same previous results, the heap space error after several logs about thetoo many levels of symbolic links.

This could be because the DWR2(directwebremoting.org) project (the project I am trying to compile) uses two different JDKs for its compilation process, please can you check this?. 
Comment 12 Stefan Bodewig 2008-10-16 03:29:02 UTC
Detecting whether a certain filesystem object is a symbolic link is surprisingly wasteful in Java (both in terms of time as well as memory required), so the code tries to avoid these checks unless it suspects it is caught in an infinite loop.

To make DirectoryScanner suspect it is in such a loop it has to see the same directory name (only the name without parents) more often than a configurable number of times.

Between the point in time where Tim confirmed his build no longer failed and the current svn version I've increased the default value of how often the same name of a directory is acceptable from one to five, see http://svn.eu.apache.org/viewvc/ant/core/trunk/src/main/org/apache/tools/ant/DirectoryScanner.java?r1=697073&r2=697692

Most likely you now hit the out-of-memory exception because Ant doesn't break out of the traversal early enough.  Could you play with the value of DirectoryScanner#MAX_LEVELS_OF_SYMLINKS to see whether a value hopefully bigger than one works for you?

Tuning this really is something we need to do before we release the code.
Comment 13 Alvaro 2008-10-16 13:47:40 UTC
Created attachment 22748 [details]
logs from compile task

The logs I get from compile (run compile task of the DWR2 project) with the max symlinks variable set to 3.
Comment 14 Alvaro 2008-10-16 13:49:52 UTC
I have checkout and compile today ant-core trunk from SVN. 

Changing MAX_LEVELS_OF_SYMLINKS=3, after 10 mins of dwr2 compile task execution
and after the --too many levels..... logs, I get this:
BUILD FAILED
java.lang.OutOfMemoryError: Java heap space
        at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:64)
        at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:75)
        at
org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1217)
        at
org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1239)
        ......



Changing MAX_LEVELS_OF_SYMLINKS=1, after some seconds of dwr2 compile task
execution and after the --too many levels..... logs, I get this:

[javac] skipping symbolic link
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/shared_bundle/Versions/CurrentJDK/Home/shared_bundle
-- too many levels of symbolic links.
    [javac] 
    [javac] 
    [javac] The system is out of resources.
    [javac] Consult the following stack trace for details.
    [javac] java.lang.OutOfMemoryError: Java heap space
    [javac]     at java.util.zip.ZipEntry.initFields(Native Method)
    [javac]     at java.util.zip.ZipEntry.<init>(ZipEntry.java:100)
    [javac]     at java.util.zip.ZipFile$3.nextElement(ZipFile.java:437)
    [javac]     at java.util.zip.ZipFile$3.nextElement(ZipFile.java:415)
    [javac]     at
com.sun.tools.javac.jvm.ClassReader.openArchive(ClassReader.java:1447)
    [javac]     at
com.sun.tools.javac.jvm.ClassReader.list(ClassReader.java:1742)
    [javac]     at
com.sun.tools.javac.jvm.ClassReader.listAll(ClassReader.java:1882)
    [javac]     at
com.sun.tools.javac.jvm.ClassReader.fillIn(ClassReader.java:1901)
    [javac]     at
com.sun.tools.javac.jvm.ClassReader.complete(ClassReader.java:1538)
    [javac]     at com.sun.tools.javac.code.Symbol.complete(Symbol.java:355)
    [javac]     at com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:256)
    [javac]     at com.sun.tools.javac.tree.Tree$TopLevel.accept(Tree.java:382)
    [javac]     at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:221)
    [javac]     at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:235)
    [javac]     at com.sun.tools.javac.comp.Enter.complete(Enter.java:460)
    [javac]     at com.sun.tools.javac.comp.Enter.main(Enter.java:445)
    [javac]     at
com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:404)
    [javac]     at com.sun.tools.javac.main.Main.compile(Main.java:592)
    [javac]     at com.sun.tools.javac.main.Main.compile(Main.java:544)
    [javac]     at com.sun.tools.javac.Main.compile(Main.java:67)
    [javac]     at com.sun.tools.javac.Main.main(Main.java:52)


I have an observation, I get this logs
[javac] skipping symbolic link
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/shared_bundle/Versions/CurrentJDK/Home/bundle
-- too many levels of symbolic links.
...
[javac] skipping symbolic link
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Versions/1.6.0/Home/bundle/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/shared_bundle
-- too many levels of symbolic links.

In the previous logs I can see references to another JDKs (1.5.0, 1.6.0) and
this type of logs are generated for all the JDKs I have installed (from 1.4 to
1.6 because the OSX updates), so why the DirectoryScanner travel all the JDKs?
(I have attached my logs file)
Comment 15 Stefan Bodewig 2008-10-17 07:24:23 UTC
you see the different JDKs because one of your symlinks is pointing to a parent directory of all your JDKs, namely the one that holds the Versions directory which again makes Ant traverse into all your JDKs.

I guess it is not so much a problem of an infinite loop but more one of getting a <path> that is incredibly big in the end and eating all your memory.

Could you perform a run with MAX_LEVELS set to 1 and run it with -debug so I stand a chance of seeing which DirectoryScanner is going into that path and why?

The defintion of the <javac> task of your buildfile would help as well.
Comment 16 Alvaro 2008-10-17 15:05:01 UTC
Created attachment 22751 [details]
DWR2 build logs

Here I attach the logs of the compile process using debug logs (-d) and MAX_LEVELS=1.
Comment 17 Alvaro 2008-10-17 15:07:35 UTC
Created attachment 22752 [details]
build.properties and build.xml compressed

Here I attach the build.properties and build.xml compressed in a zip file.
Comment 18 Alvaro 2008-10-17 15:11:01 UTC
Hi Stefan:
I have attached the project compile task log (DWR2 build logs) and the build.properties and build.xml in a compressed zip file  (build.properties and build.xml compressed).

I think that can be more productive (and easy) if you can download the DWR2 project and try yourself, (only a comment) ;) , help you is my pleasure.
Comment 19 Stefan Bodewig 2008-10-21 08:26:20 UTC
I have no idea what the DWR2 project might be.

Even if my iBook's display hadn't decided to break down a few months ago (making me an ex-Mac owner since a dell Notebook running Ubuntu was a less expensive choice), it would have been too old to run Apple's Java6 (a G3 machine).

I'm afraid without Apple's, errm, interesting *symlink* structure in JAVA_HOME I won't be able to see the same results as you.

Thanks for the logs.
Comment 20 Stefan Bodewig 2008-10-28 04:55:54 UTC
Alvaro,

in your case it is the sheer number of symbolic links together with a strange decision made by Apple about their symlinks.

If I read the logs correctly then
/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/shared_bundle
is a symbolic link pointing to
/System/Library/Frameworks/JavaVM.framework/
(this is consistent with Tim's description).

So even if Ant follows that symlink only once, it will end up scanning all your installed JVMs for jars (and each of them has a different symlink pointing to the same place).  No symlink loop detection code is going to save you from the OOM you see if there are seven VMs (1.4, 1.4.1, 1.4.2, 1.5, 1.5.0, 1.6 and 1.6.0) to recurse into.

Even with maxlevelsofsymlinks="1", the bootclasspath built-up in your compile target is huge.

I'd suggest to change your build file to read

      <bootclasspath>
        <fileset dir="${java.13.home}" includes="**/*.jar"
                 excludes="**/shared_bundle/**/>
      </bootclasspath>

instead of

      <bootclasspath>
        <fileset dir="${java.13.home}" includes="**/*.jar"/>
      </bootclasspath>

and do the same to the 1.5 javac task.

I don't think there is anything the Ant team can do, sorry.

I'm not closing this right away since I found a few places where Ant creates internal filesets where we should better restrict the number of symlinks followed, but these are unrelated to your case.
Comment 21 Stefan Bodewig 2008-10-28 05:01:01 UTC
There are no places to fix since all Filesets we create internally (in Path, for example) are not recursive (have explicit include patterns without **).

Closing again.