This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 189281 - OutOfMemoryError: Two Millions of GCed FileName objects created by parsing Api
Summary: OutOfMemoryError: Two Millions of GCed FileName objects created by parsing Api
Status: RESOLVED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 6.x
Hardware: Other Linux
: P2 normal (vote)
Assignee: Tomas Zezula
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-08-06 19:28 UTC by essotek
Modified: 2010-11-02 03:57 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description essotek 2010-08-06 19:28:25 UTC
Imported a large project and while Netbeans was scanning a directory containing a large number of Jar files an OOM error occurred.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /xx/.netbeans/6.9.1/var/log/heapdump.hprof ...
Heap dump file created [266043785 bytes in 44.703 secs]

I have gziped the heapdump down to: 76713150 bytes. Will try to attach to this defect.
Comment 1 Peter Pis 2010-08-09 07:52:14 UTC
Please use this link for attaching such a large file: http://upload.megashare.com/
Comment 2 essotek 2010-08-10 18:36:02 UTC
Heap dump is uploaded here:

http://www.megashare.com/2411524

pwd: netbeans
Comment 3 Peter Pis 2010-08-16 11:51:32 UTC
Jardo, please evaluate. For further details please contact thurka.
Comment 4 Jaroslav Tulach 2010-08-17 08:29:27 UTC
I have huge troubles to manipulate with the snapshot. For example "compute retained size" did not finish in twelve hours.

Otherwise it seems most of the space is taken by WeakHashMap and its entries (about two millions of instances). I am yet trying to figure out who is holding these entries.
Comment 5 Jaroslav Tulach 2010-08-17 08:55:18 UTC
Looks like the WeakHashMap is the one from masterfs.NamingFactory. It is supposed to contain more than a million of values, but when looking for FileName subclasses, there is just 14!
Comment 6 Jaroslav Tulach 2010-08-17 09:18:56 UTC
Tomáš H. helped me analyse the dump. The stacktraces section shows that there is a local variable of type OutOfMemoryError in RepositoryUpdater. 

at parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanBinary(:3075)
	   Local Variable: java.lang.OutOfMemoryError#8
	   Local Variable: java.util.HashMap#16379
	   Local Variable: java.util.HashMap#16380


This very likely means that the OOME happened while executing one indexBinary method. Then the finalize section of try statement catched the OOME and tried to do some clean up. The cleanup succeeded, but anyway it was too late, the OOME has been reported.

The indexBinary seems to collect all FileObject and keep them in memory. This explains while there is are WeakHashMaps with one million of entries, which are mostly null. They have been referenced while indexBinary was on stack, but as soon as the execution jumped to finalize section, they could be cleaned up.

There is not much I can do about this in filesystems. The problem is trying to hold too much fileobjects in memory. There should be some limit in the parsing.api or the scanning shall be done incrementally.
Comment 7 Tomas Zezula 2010-10-25 16:15:43 UTC
The only way how this may happen is that the user has s unarchived binary root with 2M class files.
Comment 8 Tomas Zezula 2010-10-27 15:49:32 UTC
Fixed jet-main d8452ef7b87a
I've fixed the number of active FileObjects.
The RU does not collect them but it walks the folder structure as it does before. In my opinion it's useless contract abused by some indexers. If RU does not detect changes it's OK, the indexer gets this info and does nothing. But when RU detects a change it just tells indexer "something has changed" and the indexer is responsible to find out what has changed (it has to walk the folder structure again).
I will look at other BinaryIndexers and probably do this incompatible change to improve the performance. Anyway it's not so important as class folder on class path is not very common.
Comment 9 Quality Engineering 2010-10-28 02:52:52 UTC
Integrated into 'main-golden', will be available in build *201010280000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/d8452ef7b87a
User: Tomas Zezula <tzezula@netbeans.org>
Log: #189281:OutOfMemoryError: Two Millions of GCed FileName objects created by parsing Api
Comment 10 Tomas Zezula 2010-10-29 12:28:57 UTC
I've looked at binary indexers in NB source and it seems that only JavaBinaryIndexer depends on the abused contract of isAllFilesIndexing. So I will remove the contract at all. It will take some time as I need to check performance effects on indexers.
Comment 11 Tomas Zezula 2010-11-01 16:24:02 UTC
Removed the contract as described above: jet-main 6307aa8ff816
Comment 12 Quality Engineering 2010-11-02 03:57:51 UTC
Integrated into 'main-golden', will be available in build *201011020000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/6307aa8ff816
User: Tomas Zezula <tzezula@netbeans.org>
Log: #189281:OutOfMemoryError: Two Millions of GCed FileName objects created by parsing Api