Created attachment 22327 [details] Patch to DirectoryScanner to prevent recursion on symlinks to ancestor dirs I'm using Ant 1.7.1 on Mac OS X 10.5.4 with Java SE 6 (1.6.0_05). When compiling some fairly basic Java source using the javac task I get consistent OutOfMemoryErrors. Some debugging reveals that this is due to an infinite recursion of directories in DirectoryScanner. The Java framework installed on the Mac has a Home directory that emulates the layout of a JRE on *nix/Windows, so apps (like Ant) that like to base things off $JAVA_HOME can find things. In Java 6, this Home directory has two symlinks to parent directories, namely: bundle -> ../ shared_bundle -> ../../../ (This is a default install of the latest Java update for OS X 10.5 - I haven't modified it in any way, so this issue will probably be an issue for all OS X users). During the javac execution, DirectoryScanner seems to scan the $JAVA_HOME directory (I haven't tried to ascertain why), resulting in an infinite recursion on bundle -> Home -> bundle -> Home etc. etc., eventually ending in an OOME. I'm attaching a patch that resolves listed child directories to canonical paths and checks if they're leaders of the currrent director, which fixes this issue.
I'm not convinced that ignoring the symlink completely is the right thing to do, we probably should follow it but make sure we don't scan directories we've already seen again.
Generally a lot of the internals of DirectoryScanner are private, so we can likely change signatures to pass around an e.g. Stack if needed.
looks like that won't work after all (protected scandir), but DS is written in a stateful way, so something could be done.
(In reply to comment #1) > I'm not convinced that ignoring the symlink completely is the right thing to > do, we probably should follow it but make sure we don't scan directories we've > already seen again. > It looks like DirectoryScanner already avoids scanning directories it's already encountered (via hasBeenScanned(String)). I haven't verified this actually works as intended though, and it only seems to fire on fast scans. I'd struggle to imagine a real use case for a symlink to a parent directory that I'd actually want to recurse through, but it'd be sensible if you're looking for a bullet-proof general solution.
Matt: yes, I think it is doable, even without too many or too big changes, but we need to think through some scenarios first to decide what to do. I'll take that to the dev list later (after coding up some AntUnit tests as use cases). Tim: hasBeenScanned only receives the relative path which will be different each time the directory is encountered again. I was thinking about a case where we have a structure like this A/ |---> B/ |-----> C.txt D/ |-----> E (symlink pointing to A) Let's say we set basedir to D and scan with an include pattern of E/B/C.txt - should it match the file or should we skip going upwards (we wouldn't rescan D since it isn't matched by an include pattern)? I haven't checked, but we may even get the file in fast mode and an infinite loop in slow mode (when the client asks for the not-included files/dirs). The good thing is that we don't need to think about backwards compatibility for the cases where Ant could run into an infinite loop 8-) Like I said, I'll move the discussion over to the dev list.
dev thread started http://marc.info/?l=ant-dev&m=122053158307830&w=2
the infinite loop should be gone with svn revision 693870. <fileset> now has a new attribute currently named maxLevelsOfsymlinks that currently defaults to 1. It controls how often a given symlink will be followed. Not yet documented (that's why I don't close the report) since attribute name and default value are up for discussion.
Tim, would it be possible for you to download the latest source snapshot from http://svn.apache.org/snapshots/ant/ , build it and verify that it works for you?
I downloaded and built the ant_20080922041906.tar.gz snapshot, and ran it against my build, and the compile now appears to complete successfully. I'd really like not to be spammed as much for my one offending symlink though (there's a bit of exponential explosion going on with the potential paths it gets to skip) In a probably unrelated issue, my build is now failing with the new snapshot (it appears to have broken a fairly innocuous looking AntX foreach, complaining the loop property isn't set). Log excerpt follows. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Home -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.1/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6.0/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/1.6.0/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/CurrentJDK/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bundle/Home/shared_bundle/Versions/CurrentJDK/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Home -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.1/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.1/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.2/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.4.2/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5.0/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.5.0/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6.0/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6.0/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/1.6.0/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/CurrentJDK/Home/bundle/Home/bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/CurrentJDK/Home/bundle/Home/shared_bundle -- too many levels of symbolic links. [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/shared_bundle/Versions/CurrentJDK/Home/shared_bundle -- too many levels of symbolic links. [antx:callforeach] antx:callforeach unable to run all steps; step [copy.inline.resources] threw failure. [antx:callforeach] Unable to complete steps when loop cursor at [/Users/archie/Projects/OrionPlatform/Tools/com.orchestral.common.hibernate3/src/main/java].
more recent versions will allow the same symlink to be traversed more often (to reduce the performance and memory impact of the tests), so your build will consume a bit more memory. We might lower the log priority, by writing the message to System.out, but it would probably remain visible anyway. I'm afraid there isn't much we can do.
I have checkout the trunk from SVN (october 14) compile it and tried to compile a project and the problem is present yet. I have the same logs as in the Comment #9, with the same previous results, the heap space error after several logs about thetoo many levels of symbolic links. This could be because the DWR2(directwebremoting.org) project (the project I am trying to compile) uses two different JDKs for its compilation process, please can you check this?.
Detecting whether a certain filesystem object is a symbolic link is surprisingly wasteful in Java (both in terms of time as well as memory required), so the code tries to avoid these checks unless it suspects it is caught in an infinite loop. To make DirectoryScanner suspect it is in such a loop it has to see the same directory name (only the name without parents) more often than a configurable number of times. Between the point in time where Tim confirmed his build no longer failed and the current svn version I've increased the default value of how often the same name of a directory is acceptable from one to five, see http://svn.eu.apache.org/viewvc/ant/core/trunk/src/main/org/apache/tools/ant/DirectoryScanner.java?r1=697073&r2=697692 Most likely you now hit the out-of-memory exception because Ant doesn't break out of the traversal early enough. Could you play with the value of DirectoryScanner#MAX_LEVELS_OF_SYMLINKS to see whether a value hopefully bigger than one works for you? Tuning this really is something we need to do before we release the code.
Created attachment 22748 [details] logs from compile task The logs I get from compile (run compile task of the DWR2 project) with the max symlinks variable set to 3.
I have checkout and compile today ant-core trunk from SVN. Changing MAX_LEVELS_OF_SYMLINKS=3, after 10 mins of dwr2 compile task execution and after the --too many levels..... logs, I get this: BUILD FAILED java.lang.OutOfMemoryError: Java heap space at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:64) at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:75) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1217) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1239) ...... Changing MAX_LEVELS_OF_SYMLINKS=1, after some seconds of dwr2 compile task execution and after the --too many levels..... logs, I get this: [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/shared_bundle/Versions/CurrentJDK/Home/shared_bundle -- too many levels of symbolic links. [javac] [javac] [javac] The system is out of resources. [javac] Consult the following stack trace for details. [javac] java.lang.OutOfMemoryError: Java heap space [javac] at java.util.zip.ZipEntry.initFields(Native Method) [javac] at java.util.zip.ZipEntry.<init>(ZipEntry.java:100) [javac] at java.util.zip.ZipFile$3.nextElement(ZipFile.java:437) [javac] at java.util.zip.ZipFile$3.nextElement(ZipFile.java:415) [javac] at com.sun.tools.javac.jvm.ClassReader.openArchive(ClassReader.java:1447) [javac] at com.sun.tools.javac.jvm.ClassReader.list(ClassReader.java:1742) [javac] at com.sun.tools.javac.jvm.ClassReader.listAll(ClassReader.java:1882) [javac] at com.sun.tools.javac.jvm.ClassReader.fillIn(ClassReader.java:1901) [javac] at com.sun.tools.javac.jvm.ClassReader.complete(ClassReader.java:1538) [javac] at com.sun.tools.javac.code.Symbol.complete(Symbol.java:355) [javac] at com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:256) [javac] at com.sun.tools.javac.tree.Tree$TopLevel.accept(Tree.java:382) [javac] at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:221) [javac] at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:235) [javac] at com.sun.tools.javac.comp.Enter.complete(Enter.java:460) [javac] at com.sun.tools.javac.comp.Enter.main(Enter.java:445) [javac] at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:404) [javac] at com.sun.tools.javac.main.Main.compile(Main.java:592) [javac] at com.sun.tools.javac.main.Main.compile(Main.java:544) [javac] at com.sun.tools.javac.Main.compile(Main.java:67) [javac] at com.sun.tools.javac.Main.main(Main.java:52) I have an observation, I get this logs [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Versions/1.5.0/Home/shared_bundle/Versions/CurrentJDK/Home/bundle -- too many levels of symbolic links. ... [javac] skipping symbolic link /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Home/bundle/Home/bundle/Home/bundle/Home/shared_bundle/Versions/1.6.0/Home/bundle/Home/bundle/Home/shared_bundle/Versions/1.4.2/Home/shared_bundle -- too many levels of symbolic links. In the previous logs I can see references to another JDKs (1.5.0, 1.6.0) and this type of logs are generated for all the JDKs I have installed (from 1.4 to 1.6 because the OSX updates), so why the DirectoryScanner travel all the JDKs? (I have attached my logs file)
you see the different JDKs because one of your symlinks is pointing to a parent directory of all your JDKs, namely the one that holds the Versions directory which again makes Ant traverse into all your JDKs. I guess it is not so much a problem of an infinite loop but more one of getting a <path> that is incredibly big in the end and eating all your memory. Could you perform a run with MAX_LEVELS set to 1 and run it with -debug so I stand a chance of seeing which DirectoryScanner is going into that path and why? The defintion of the <javac> task of your buildfile would help as well.
Created attachment 22751 [details] DWR2 build logs Here I attach the logs of the compile process using debug logs (-d) and MAX_LEVELS=1.
Created attachment 22752 [details] build.properties and build.xml compressed Here I attach the build.properties and build.xml compressed in a zip file.
Hi Stefan: I have attached the project compile task log (DWR2 build logs) and the build.properties and build.xml in a compressed zip file (build.properties and build.xml compressed). I think that can be more productive (and easy) if you can download the DWR2 project and try yourself, (only a comment) ;) , help you is my pleasure.
I have no idea what the DWR2 project might be. Even if my iBook's display hadn't decided to break down a few months ago (making me an ex-Mac owner since a dell Notebook running Ubuntu was a less expensive choice), it would have been too old to run Apple's Java6 (a G3 machine). I'm afraid without Apple's, errm, interesting *symlink* structure in JAVA_HOME I won't be able to see the same results as you. Thanks for the logs.
Alvaro, in your case it is the sheer number of symbolic links together with a strange decision made by Apple about their symlinks. If I read the logs correctly then /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home/shared_bundle is a symbolic link pointing to /System/Library/Frameworks/JavaVM.framework/ (this is consistent with Tim's description). So even if Ant follows that symlink only once, it will end up scanning all your installed JVMs for jars (and each of them has a different symlink pointing to the same place). No symlink loop detection code is going to save you from the OOM you see if there are seven VMs (1.4, 1.4.1, 1.4.2, 1.5, 1.5.0, 1.6 and 1.6.0) to recurse into. Even with maxlevelsofsymlinks="1", the bootclasspath built-up in your compile target is huge. I'd suggest to change your build file to read <bootclasspath> <fileset dir="${java.13.home}" includes="**/*.jar" excludes="**/shared_bundle/**/> </bootclasspath> instead of <bootclasspath> <fileset dir="${java.13.home}" includes="**/*.jar"/> </bootclasspath> and do the same to the 1.5 javac task. I don't think there is anything the Ant team can do, sorry. I'm not closing this right away since I found a few places where Ant creates internal filesets where we should better restrict the number of symlinks followed, but these are unrelated to your case.
There are no places to fix since all Filesets we create internally (in Path, for example) are not recursive (have explicit include patterns without **). Closing again.