Bug 41280 - APT Task doesn't fail if forked process dies with an OutOfMemoryException
Summary: APT Task doesn't fail if forked process dies with an OutOfMemoryException
Status: REOPENED
Alias: None
Product: Ant
Classification: Unclassified
Component: Core tasks (show other bugs)
Version: 1.7.0
Hardware: Other other
: P2 normal (vote)
Target Milestone: 1.7.1
Assignee: Ant Notifications List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-01-02 12:45 UTC by David Ehrlich
Modified: 2009-07-10 08:11 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Ehrlich 2007-01-02 12:45:52 UTC
While processing a large number of files via an 'apt' ant task, apt is failing
with an out of memory exception.  Altering the task to include a
'memoryMaximumSize' argument does not rectify the situation.  The following is
emitted regardless of the 'fork=' setting in the ant script:

      [apt] Compiling 1881 source files
      [apt] Since fork is false, ignoring memoryMaximumSize setting.
      [apt]
      [apt]
      [apt] The system is out of resources.
      [apt] Consult the following stack trace for details.
      [apt] java.lang.OutOfMemoryError: Java heap space

I haven't found a way to work around the issue via the script.  However, I was
able to get the Apt task to do what I need by altering the
org.apache.tools.ant.taskdefs.Apt constructor as follows.

Original constructor:
    public Apt() {
        super();
        super.setCompiler(AptExternalCompilerAdapter.class.getName());
        setFork(true);
    }
Work-around constructor:
    public Apt() {
        super();
        super.setCompiler(AptExternalCompilerAdapter.class.getName());
        super.setFork(true);
    }

However, the workaround causes the task to emit a different warning:
      [apt] Since compiler setting isn't classic or modern,ignoring fork setting.
      [apt] Compiling 1881 source files
      [apt] Since compiler setting isn't classic or modern,ignoring fork setting.

I suspect this implies other changes need to be made to get the apt task to
recognize the appropriate memory options.
Comment 1 Steve Loughran 2007-01-03 05:56:19 UTC
good catch. we turned forking off because we want people to fork every time
(classpath problems) but in fact hadn't fixed the problem because we'd stubbed
out the setFork() operator. 

Expect a fix for Ant1.7.1
Comment 2 Steve Loughran 2007-01-03 13:48:16 UTC
fixed in SVN, you can check out and use the new version or stick with your
patched edition until ant1.7.1 ships. Thank you for finding a bug that snuck
past our testing!
Comment 3 Marek Slama 2008-10-31 03:18:49 UTC
It happens again even with ant 1.7.1 with 64bit JDK 5u16 on Ubuntu Intrepid. I am not able to say if it is specific to ant package on Ubuntu.
[local]> ant -version
Apache Ant version 1.7.1 compiled on October 3 2008

  <target name="compile-jruby" depends="compile-tasks, compile-annotation-binder, check-for-optional-packages">
    <!-- Generate binding logic ahead of time -->
    <apt factory="org.jruby.anno.AnnotationBinder" destdir="${jruby.classes.dir}" debug="true" source="${javac.version}" target="${javac.version}" deprecation="true" encoding="UTF-8">
      <classpath refid="build.classpath"/>
      <classpath path="${jruby.classes.dir}"/>
      <src path="${src.dir}"/>
      <patternset refid="java.src.pattern"/>
      <compilerarg line="-XDignore.symbol.file=true"/>
    </apt>
  </target>


compile-jruby:
      [apt] Since compiler setting isn't classic or modern,ignoring fork setting.
      [apt] Compiling 641 source files to /mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build/classes/jruby
      [apt] Since compiler setting isn't classic or modern,ignoring fork setting.
      [apt] warning: Annotation types without processors: [java.lang.Override, java.lang.SuppressWarnings, java.lang.Deprecated, org.jruby.anno.JRubyConstant, org.jruby.anno.JRubyModule, java.lang.annotation.Retention, java.lang.annotation.Target]
      [apt] Problem encountered during annotation processing; 
      [apt] see stacktrace below for more information.
      [apt] java.lang.OutOfMemoryError: Java heap space
      [apt] 	at java.io.BufferedWriter.<init>(BufferedWriter.java:87)
      [apt] 	at java.io.BufferedWriter.<init>(BufferedWriter.java:70)
      [apt] 	at java.io.PrintStream.init(PrintStream.java:83)
      [apt] 	at java.io.PrintStream.<init>(PrintStream.java:100)
      [apt] 	at java.io.PrintStream.<init>(PrintStream.java:62)
      [apt] 	at org.jruby.anno.AnnotationBinder$AnnotationBindingProcessor$RubyClassVisitor.visitClassDeclaration(AnnotationBinder.java:90)
      [apt] 	at com.sun.tools.apt.mirror.declaration.ClassDeclarationImpl.accept(ClassDeclarationImpl.java:95)
      [apt] 	at com.sun.mirror.util.DeclarationScanner.visitClassDeclaration(DeclarationScanner.java:110)
      [apt] 	at com.sun.tools.apt.mirror.declaration.ClassDeclarationImpl.accept(ClassDeclarationImpl.java:95)
      [apt] 	at com.sun.mirror.util.DeclarationScanner.visitClassDeclaration(DeclarationScanner.java:125)
      [apt] 	at com.sun.tools.apt.mirror.declaration.ClassDeclarationImpl.accept(ClassDeclarationImpl.java:95)
      [apt] 	at org.jruby.anno.AnnotationBinder$AnnotationBindingProcessor.process(AnnotationBinder.java:60)
      [apt] 	at com.sun.mirror.apt.AnnotationProcessors$CompositeAnnotationProcessor.process(AnnotationProcessors.java:60)
      [apt] 	at com.sun.tools.apt.comp.Apt.main(Apt.java:454)
      [apt] 	at com.sun.tools.apt.main.JavaCompiler.compile(JavaCompiler.java:448)
      [apt] 	at com.sun.tools.apt.main.Main.compile(Main.java:1075)
      [apt] 	at com.sun.tools.apt.main.Main.compile(Main.java:938)
      [apt] 	at com.sun.tools.apt.Main.processing(Main.java:95)
      [apt] 	at com.sun.tools.apt.Main.process(Main.java:43)
      [apt] 	at com.sun.tools.apt.Main.main(Main.java:34)
      [apt] Note: Some input files use unchecked or unsafe operations.
      [apt] Note: Recompile with -Xlint:unchecked for details.

[o.jruby.distro]> java -version
java version "1.5.0_16-rev"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-rev-b09)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_16-rev-b09, mixed mode)
[o.jruby.distro]> javac -version
javac 1.5.0_16-rev
Comment 4 Marek Slama 2008-10-31 03:20:54 UTC
I forgot to say: The same target passes on 32bit JDK.
Comment 5 Steve Loughran 2008-10-31 04:21:29 UTC
It probably only fails on a 64 bit VM because Sun 64-bit JVM heap requirements are 1.5X 32 bit due entirely to memory pointer length; it probably works on the JRockit JVM which uses short pointers when the heap is <4GB.

What is the ant -verbose run?

The "no forking" message implies that Apt is running in Ant's own process, so if you set the heap size there then it should propagate down. That would make the problem go away, but not fix any defect. 

Comment 6 Marek Slama 2008-10-31 04:48:57 UTC
I set ANT_OPTS to "-Xmx1024m -Dbuild.compiler.debug=true" but it does not help anyway. It would say this should be enough so there might be another problem specific to 64bit JDK. Verbose output for this target (I build o.jruby.distro module from NetBeans sources.) is ~5MB. If you wish I can send zip file

Hopefully relevant part is (let me know if you want anything else):
      [apt] Since compiler setting isn't classic or modern,ignoring fork setting.
      [apt] Compiling 641 source files to /mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build/classes/jruby
      [apt] Using external apt compiler
      [apt] Since compiler setting isn't classic or modern,ignoring fork setting.
      [apt] Compilation arguments:
      [apt] '-deprecation'
      [apt] '-d'
      [apt] '/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build/classes/jruby'
      [apt] '-classpath'
      [apt] '/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build/classes/jruby:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/asm-3.0.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/asm-analysis-3.0.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/asm-commons-3.0.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/asm-tree-3.0.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/asm-util-3.0.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/backport-util-concurrent.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/bnd-0.0.249.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/bytelist-0.1.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/dynalang-0.3.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/emma.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/emma_ant.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/jarjar-1.0rc7.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/jline-0.9.93.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/jna-posix.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/jna.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/joda-time-1.5.1.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/joni.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/junit.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/jvyamlb-0.2.3.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/retroweaver-2.0.5.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/build_lib/retroweaver-rt-2.0.5.jar:/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/lib/bsf.jar:/usr/share/ant/lib/ant-launcher.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/xercesImpl.jar:/usr/share/ant/lib/ant-javamail.jar:/usr/share/ant/lib/ant-apache-oro.jar:/usr/share/ant/lib/ant-commons-logging.jar:/usr/share/ant/lib/ant-apache-bsf.jar:/usr/share/ant/lib/ant-apache-regexp.jar:/usr/share/ant/lib/ant-nodeps.jar:/usr/share/ant/lib/ant-jsch.jar:/usr/share/ant/lib/ant-jmf.jar:/usr/share/ant/lib/ant-apache-log4j.jar:/usr/share/ant/lib/ant.jar:/usr/share/ant/lib/ant-commons-net.jar:/usr/share/ant/lib/ant-apache-resolver.jar:/usr/share/ant/lib/ant-antlr.jar:/usr/share/ant/lib/ant-apache-bcel.jar:/usr/share/ant/lib/ant-trax.jar:/usr/share/ant/lib/ant-junit.jar:/usr/share/ant/lib/ant-jdepend.jar:/usr/share/ant/lib/ant-swing.jar:/usr/java/jdk1.5.0_16_64bit/lib/tools.jar'
      [apt] '-sourcepath'
      [apt] '/mnt/local/mslama/netbeans/hg-nbsrc/core-main/o.jruby.distro/unpatched_source/jruby-1.1.4/src'
      [apt] '-target'
      [apt] '1.5'
      [apt] '-encoding'
      [apt] 'UTF-8'
      [apt] '-g'
      [apt] '-XDignore.symbol.file=true'
      [apt] '-source'
      [apt] '1.5'
      [apt] '-factory'
      [apt] 'org.jruby.anno.AnnotationBinder'
      [apt]
      [apt] The ' characters around the executable and arguments are
      [apt] not part of the command.
      [apt] Files to be compiled:
Comment 7 Steve Loughran 2008-10-31 05:15:14 UTC
looking at the source, that warning about forking is invalid; its the superclass whining. The stack trace implies this is a new JVM (no ant classes in the trace) so we are forking. That leaves the question of whether the memory options are going down to the process or not.

Can you now do an ant -debug run and paste in the [apt] bit of that log?
Comment 8 Marek Slama 2008-10-31 05:49:29 UTC
I used memoryMaximumSize="256m" for apt task and it works now. Yes warning about forking is confusing and should be eliminated. Parent process memory settings are not propagated to forked apt task so maximum memory size must be increased explicitly. But it is documented to it is correct. Thank you for help. I assume ant -debug output is not necessary anymore. If you still need it let me know.
Comment 9 Marek Slama 2008-10-31 05:56:55 UTC
Problem is that task does not fail when OutOfMemoryError happens in forked JVM. Is it correct? It makes diagnostic difficult in case of building big project.