This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 258638 - Parsing huge C source file leads to GC limit exceed, sometimes NB hangs up
Summary: Parsing huge C source file leads to GC limit exceed, sometimes NB hangs up
Status: RESOLVED WORKSFORME
Alias: None
Product: cnd
Classification: Unclassified
Component: Code Model (show other bugs)
Version: 8.1
Hardware: PC Windows 7
: P2 normal (vote)
Assignee: Vladimir Voskresensky
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-04-04 05:56 UTC by mariusz_m
Modified: 2016-04-06 06:13 UTC (History)
0 users

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
out of mem - messages log (46.91 KB, application/octet-stream)
2016-04-05 12:41 UTC, mariusz_m
Details

Note You need to log in before you can comment on or make changes to this bug.
Description mariusz_m 2016-04-04 05:56:06 UTC
In my embedded project I have no file system, so test data is compiled into the code. I have simple C file containing 
const unsigned char initial_image[1200*960] = { 0x01, 0xc1, 0xff, ... } 
- the file is over 50MB in length.

Every time I open the project I have the error below. 

I understand the file is unusually big, but in this case please declare some file size limit (like for the syntax parser: > 4000 lines of code only basic syntax) above which the project structure parser will simply skip the file. I already have very big Java stack size for NB, making it bigger has no sense to me. 
BTW in parallel I'm using Eclipse based IDE for the target CPU - it has no problems with such a big files in the project (of course I never open it in the editor :) )

Or maybe I'm wrong and file size has nothing to do with it, as for me the parser has pretty nothing to do here as there is only one single array and no includes!

~~
Mariusz

***************

java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at com.sun.jna.Function.isVarArgs(Function.java:782)
	at com.sun.jna.Function.invoke(Function.java:292)
	at com.sun.jna.Library$Handler.invoke(Library.java:212)
	at org.netbeans.modules.masterfs.watcher.windows.$Proxy2.ReadDirectoryChangesW(Unknown Source)
	at org.netbeans.modules.masterfs.watcher.windows.WindowsNotifier.handleChanges(WindowsNotifier.java:455)
	at org.netbeans.modules.masterfs.watcher.windows.WindowsNotifier.access$200(WindowsNotifier.java:77)
	at org.netbeans.modules.masterfs.watcher.windows.WindowsNotifier$2.run(WindowsNotifier.java:403)
Comment 1 Alexander Simon 2016-04-04 15:03:15 UTC
Please provide additional info:
- IDE message log
- Heap dump
Comment 2 mariusz_m 2016-04-05 06:38:53 UTC
What? Heap dump? I don't understand what you talking about. I'm only user, not NB developer ;)
Beside, if my description is not enough to reproduce, it means the problem with parsing big files does not exists on your side, does it ?
Comment 3 Alexander Simon 2016-04-05 07:25:39 UTC
(In reply to mariusz_m from comment #2)
> What? Heap dump? I don't understand what you talking about. I'm only user,
> not NB developer ;)
> Beside, if my description is not enough to reproduce, it means the problem
> with parsing big files does not exists on your side, does it ?
Hi,

Unfortunately your problem description is not enough to understand a problem.
At first we need an IDE message log (http://wiki.netbeans.org/FaqLogMessagesFile).
Java heap dump (http://wiki.netbeans.org/FaqMemoryDump) makes possible to analyze root cause of memory lack.
Instead of heap dump (it is very big) you can attach a threads dump (http://wiki.netbeans.org/GenerateThreadDump). Probably it will be enough.

Of course, IDE has a problem with parsing huge files. To solve such problem IDE parser has a limit of file token stream (as i remember 8,000,000 tokens and IDE will consume about 1g to parse such file).

You can try ti increase java heap size by option:
-J-Xmx3g
Comment 4 mariusz_m 2016-04-05 12:41:14 UTC
Created attachment 159121 [details]
out of mem - messages log
Comment 5 mariusz_m 2016-04-05 13:02:16 UTC
Please find the heap dump here:
https://drive.google.com/open?id=0B_TIY3RAz3G0dHA4YXBhdG1CZnM

It's 80MB, uncompressed 400MB.
Comment 6 Alexander Simon 2016-04-05 15:59:42 UTC
I see Java option -Xmx512m.
I'm sure that IDE will have a lack of memory with 512Mb heap memory.
I'd recommend to use at least -J-Xmx2g

According to heap dump the biggest objects:
150Mb in 1,943 instances of org.clang.lex.SmallVectorToken
120Mb in 2,497,407 instances of org.clang.lex.Token
Comment 7 Vladimir Voskresensky 2016-04-05 16:11:54 UTC
(In reply to Alexander Simon from comment #6)
> I see Java option -Xmx512m.
> I'm sure that IDE will have a lack of memory with 512Mb heap memory.
> I'd recommend to use at least -J-Xmx2g
> 
> According to heap dump the biggest objects:
> 150Mb in 1,943 instances of org.clang.lex.SmallVectorToken
> 120Mb in 2,497,407 instances of org.clang.lex.Token
These 120 are included in the above 150.

So, the problem is with default Xmx which was assigned to JVM for running NB (512Mb).
You can control it in $NB/etc/netbeans.conf file 
http://wiki.netbeans.org/FaqSettingHeapSize
I would recommend to put there -J-Xmx2G and you will not have problems
Comment 8 mariusz_m 2016-04-06 06:13:05 UTC
Hi,

setting maximum heap to 2GB helped - no more exception during opening the project.
Also, setting initial heap size to 1GB instead of 32MB greatly improved the startup time :)

For the future: 
How about suggesting the user to modify these setting in case of error?
Extend this manual
https://performance.netbeans.org/howto/jvmswitches/index.html
and point it when out of memory exception happened ? This will save your time.

Thanks, Mariusz