Bug 202672

Created attachment 111189 [details]
Self-Profiling

Looking at the snapshot.
The IDE had not enough memory to do the compilation of some (one) root by fast OnePassCompileWorker, it had to use the slow MultiPassCompileWorker (the fallback when there is not enough memory).
The MultiPassCompileWorker took 76.5% of the indexing time.
Increase the IDE heap to improve the indexing speed.

Also it's strange that java.io.File.close is taking 11% of the scan time, is it a local disk?
What is the IDE heap size?
How big is the biggest source root? I can open big roots like the OpenJDK with default Xmx settings without fallback to MultiPassCompileWorker.

(In reply to comment #2)
> Increase the IDE heap to improve the indexing speed.
My Config-Settings:
"-J-Xss2m -J-Xms32m -J-XX:PermSize=32m -J-Dnetbeans.logger.console=true -J-ea -J-Dapple.laf.useScreenMenuBar=true -J-Dapple.awt.graphics.UseQuartz=true -J-Dsun.java2d.noddraw=true -J-Dsun.zip.disableMemoryMapping=true -J-Dplugin.manager.check.updates=false -J-Xmx1G -J-XX:+UseConcMarkSweepGC -J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled -J-XX:+UseCompressedOops -J-Djava.index.useMemCache=true"

I think the Xmx is enough?

> Also it's strange that java.io.File.close is taking 11% of the scan time, is it
> a local disk?
It´s a local SSD.

> What is the IDE heap size?
1G

> How big is the biggest source root? 
~12000 Java-Files.

Can you please try

1. Even increase the heap (to say 1.5 G)
2. to run 7.1 RC1 build (after it gets published on Monday)?
3. (I know this might not be possible but still wanted to mention it) split your project to couple of more smaller projects?

Thanks, David

The 12000 java files are in single source root?
The IDE first tries to compile the files by OnePassCompileWorker (OPCW) if possible.
If there is not enough memory to do it, the IDE uses MultiPassCompileWorker which is
much slower (as seen even from name), unfortunately it cannot be made faster. It takes file
by file parses it, attributes it (but attribution requires another source files, so it has to parse
them at the same point it gets into state that it has not enough of memory, but it has already
attributed some dependent files, so it dumps them into disk, frees memory and parses and attributes
the file again (now with data from caches which were dumped).

The strange is that lots of time is spent in File.close() mot sure if it's SSD specific, FS specific or it may be an antivirus.

Created attachment 113697 [details]
Message-Log + NPS-Files of the different Settings.

I made the following Test:
-) Cleared Cache-Directory
-) Started NetBeans on Win7x64
-) Pressed the Selfprofiling-Button and then I was waiting while Scanning-Projects has finished.

Result:
-) Java 1.6 Xmx=1500M
    Scanning took: 364133ms
-) Java 1.6 Xmx=1000M
    Scanning took: 566318ms
-) Java 1.7 Xmx=1500M
    Scanning took: 419185ms
-) Java 1.7 Xmx=1000M
    Scanning took: 459419ms

The measurement proved what I've written, while compiling huge source roots (single source root containing 12000 source files) adding memory improves the speed of scan.
Especially visible on JDK 1.6:

-) Java 1.6 Xmx=1500M
    Scanning took: 364133ms
-) Java 1.6 Xmx=1000M
    Scanning took: 566318ms

The adding of memory caused decrease of scan by more then 40% which is fine (not a bug it's a feature).
If you have huge monolithic source root you can either add memory or decompose it into more source roots (projects) otherwise the IDE will try to do the best to handle such a root without OOM but you cannot expect it to be very fast because of the way it has to solve it (described in comment #5)

For the JDK 7 the slow and fast path move closer, caused by GC settings, I will look at the snapshots which strategy was used in case of JDK 7.0 and let you know.

For big monolithic source root you need to increase heap to to make the compilation faster.
Unfortunately the IDE does not tell it to you. In case when IDE falls into slow MultiPassCompileWorker it should display notification with link on wiki page containing info how to increase the heap.