Bug 35476 - XMLLogger causes OutOfMemoryError
Summary: XMLLogger causes OutOfMemoryError
Status: RESOLVED DUPLICATE of bug 46058
Alias: None
Product: Ant
Classification: Unclassified
Component: Core (show other bugs)
Version: 1.6.5
Hardware: Other other
: P2 critical with 1 vote (vote)
Target Milestone: ---
Assignee: Ant Notifications List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-06-23 07:29 UTC by Juergen Melzer
Modified: 2008-10-21 21:11 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Juergen Melzer 2005-06-23 07:29:20 UTC
I use the XMLLogger when starting ANT. The build process takes a long time and 
makes many output. After a while will crashes with OutOfMemory. 
When I don't use the XMLLogger everything works fine.

When I take a look in the logger I see, that the logger flush only when the 
build finished.
I mean it will solves the problem when the DOM is written every X event, isn't 
it?
Comment 1 J.M. (Martijn) Kruithof 2005-06-23 18:52:42 UTC
No in that case SAX should be used instead of DOM
Comment 2 Juergen Melzer 2005-06-26 19:10:04 UTC
I am not interested how the solution will be I am interested in a solution.
So you mean it is a bug or not?
Comment 3 Phil Weighill-Smith 2005-06-26 19:56:14 UTC
Seems to me not a bug but rather an design flaw for "large" builds. Also, a 
little more care should be applied to how you phrase your updates so they don't 
sound rude.
Comment 4 Alexey Solofnenko 2005-06-26 20:15:56 UTC
This is "as-designed" behaviour. SAX will not help - it is only for XML reading.
XMLLogger would have to be rewritten not to create DOM document in memory, 
but to incrementally output XML as text directly into a file. If you are in a
hurry, 
just write your own logger. You can look at the current XMLLogger implementation
in ANT source distribution, in FishEye: 
http://www.cenqua.com/fisheye/demo/viewrep/ant/src/main/org/apache/tools/ant/XmlLogger.java


- Alexey.
Comment 5 Alexey Solofnenko 2005-06-26 20:17:31 UTC
... or just increase your ANT JVM maximum memory size.
Comment 6 Juergen Melzer 2005-06-26 20:35:57 UTC
Sorry for the sound.
I give the process about 2 GB.
Comment 7 J.M. (Martijn) Kruithof 2005-06-26 20:37:07 UTC
Strictly speaking this is not a bug, but a limitation.

The problem will in that case be that for the attributes to be written one must
wait with writing the opening tag until all attributes are known, but how to
know if all attributes of an opening tag are known, without waiting until the
element is closed? In this case still all data would have to be buffered until
the build is finished.

This could by solved of course by using another XML schema, that only uses
nested elements, and not attributes ( or attributes on a special attributes element)

like

<output>
<nested>
somedata
<attributes name="value"/>
</nested>
<nested>
<attributes name="value"/>
</nested>
<nested>
</nested>
<attributes name="value"/>
</output>

Obviously this would severely impact the XML schema (nothing that cannot be
solved by processing using XSLT).
Comment 8 Alexey Solofnenko 2005-06-26 20:43:04 UTC
We can try writing all elements as soon as they are closed instead of waiting
for the build to finish.
Comment 9 J.M. (Martijn) Kruithof 2005-06-26 20:54:54 UTC
Only if we log the attributes of the build element in an alternative way. (The
attributes are not known until the build is finished.)
Comment 10 J.M. (Martijn) Kruithof 2005-06-26 21:02:37 UTC
By the way, this would also be valid for container tasks (like parallel or
sequential).
Comment 11 Alexey Solofnenko 2005-06-26 21:28:51 UTC
I agree with you. To make the output incremental, the output format has to be
changed.
Comment 12 Jose Alberto Fernandez 2005-06-27 14:11:20 UTC
OK, is it every attribute that is unknown at the begining on the startXXX or 
just certain things like "elapsedTime" and "errorStack" (paraphasing).

Maybe we can just move the things that are really lookaheads to their correct 
place and make the logger generate SAX events. By using SAX, we gain the fact 
that we can pipe the events to a stylesheet XSLT, if you do not mind DOM, or 
STX if you want to continue with streamming (using joost). 

This would be way better. The only thing you need is a way to make sure all 
elements get close after a build fails so that we always get correct XML. I 
may have some XML SAX filter code that knkows how to do that. If needed.
Comment 13 Steve Loughran 2005-07-04 13:39:54 UTC
Well, there is nothing to prevent us from changing the generated XML and the
stylesheets. What may break is anyone using their own stylesheets.

For reference, in the distributed JUnit stuff I've done (I hope to demo this at
apachecon soon), I dont use dom/SAX; I just generate simple XML by hand,
escaping stuff where needed. It's cheap and limited, but works well.

The problem I had with the existing junit stuff was not memory consumption but
robustness: I wanted as much of the XML data written to file, so that if
something crashed, we would have more of a trace.

In ant, we stick the statistics at the top:
        rootElement.setAttribute(ATTR_TESTS, "" + suite.runCount());
        rootElement.setAttribute(ATTR_FAILURES, "" + suite.failureCount());
        rootElement.setAttribute(ATTR_ERRORS, "" + suite.errorCount());
        rootElement.setAttribute(ATTR_TIME, "" + (suite.getRunTime() / 1000.0));
whereas in my (custom) junit logging, I add a new element at the end of the run

        write("summary",
                a("tests", testCount)
                +
                a("failures", failureCount)
                + a("errors", errorCount),
                null, false);

        xmlFile.write(ROOT_CLOSE);
This leaves something like
<summary tests="4" failures="5" errors="0" ></summary>
at the end of the document.
Comment 14 Juergen Melzer 2005-07-04 22:00:59 UTC
So I use Cruisecontrol too. And when you change the XML-Format the Cruisecontrol
XSLT will not work anymore, So be careful of changing the format.
Comment 15 Alexey Solofnenko 2005-07-05 00:18:08 UTC
Maybe instead of updating old logger, it is better to create a new one? In that case we will preserve backward compatibility.
Comment 16 Jozef Hovan 2005-07-15 11:22:14 UTC
See bug 35750, I guess, you don't need to rewrite XMLLogger.
Comment 17 J.M. (Martijn) Kruithof 2005-07-16 14:34:17 UTC
The amount of memory needed for the non-closed classloaders under certain
circumstances in the junit test task are orders of magnitude lower than the data
kept for logging in the XMLLogger. This is a problem of its own.
Comment 18 Brett Adam 2005-10-21 16:42:23 UTC
The fact that the XMLLogger keeps the entire log DOM in memory until the end 
of the Ant run causes MASSIVE problems for anyone using Ant to run large build 
scripts. We are seeing Ant logs that are routinely 30MB in size, which consume 
enormous (read: > 800MB) of memory during execution.

Please consider writing the DOM out as it emerges even if that means the 
intermediate XML output file is invalid XML unti lthe run is over, and even if 
it means aving to disable certain "features" like execution timing stats on 
each nested node.
Comment 19 Stefan Bodewig 2008-10-21 21:11:29 UTC
I don't think we can solve this within the constraints of the XmlLogger format - or more precisely while using XML at all.  The fact that targets could be run in parallel means we couldn't write any XML tag that was somehow nested inside a target incrementally at all.

The only solution can be a different logger IMHO.

*** This bug has been marked as a duplicate of bug 46058 ***