Bug 221331

It seems indexing infrastructure unusable for mozilla project.

see also bug #221342 and bug #221337

P2 because:
- IDE was freezed more then 30 minutes
- IDE eat 2G memory

Please attach IDE self profile, according to the linked issue the CSS & HTML parsing is done which has not much common with general p.a. There is already an issue about expensive HTML validation https://netbeans.org/bugzilla/show_bug.cgi?id=206025.

Created attachment 127011 [details]
Beggining of indexing

Created attachment 127013 [details]
just before IDE hung. IDE have eaten 2G memory

Thanks!

CSS parsing.
If it's possible can you attach a longer snapshot, for example 2 or 5 mins.
In the snapshot there is only CSS parsing but not HTML validator which I expect to be there as well.
Thanks!

(In reply to comment #8)
> CSS parsing.
> If it's possible can you attach a longer snapshot, for example 2 or 5 mins.
> In the snapshot there is only CSS parsing but not HTML validator which I expect
> to be there as well.
> Thanks!
Unfortunately I cant because IDE died (not responsible).

>Wait 10 minutes
Can you try to to start sampling a bit earlier?
After 5 mins?

(In reply to comment #10)
> >Wait 10 minutes
> Can you try to to start sampling a bit earlier?
> After 5 mins?
I give IDE 2G heap.
IDE start indexing.
I start profiling.
When IDE eat 1900M memory I take first snapshot.
After that IDE becames partly unresponsible, but I managed to take short second snapshot before IDE died.

Sorry, my fault I've overlooked the first snapshot.
That's exactly what I wanted.
Thanks!

Looking at the first snapshot the most time is spent in indexing javascript.
Seems a memory leak in the javascript as it takes 1900M.
I will clone mozilla central and generate heap dump to find out more.

Tomas thanks for the help. I don't have the Mozilla central and I'm going to stop in your office tomorrow.

#mkdir firefox
#cd firefox
#export CVSROOT=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
#cvs checkout mozilla/client.mk
#cd mozilla 

create .mozconfig file with the following content:
mk_add_options MOZ_CO_PROJECT=browser
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/firefox
ac_add_options --enable-extensions=default,webservices
ac_add_options --enable-application=browser
ac_add_options --enable-debug

#make -f client.mk checkout 

Then extract inside top firefox folder (on the same level with mozilla folder) attached nb project

Created attachment 127239 [details]
NB project for firefox

To turn on TODO support (which activates js parsing) you should Tools->Options->C/C++->Other and set 
"Support indexing of non C/C++/Fortran files in projects (slow down parsing)"

Thanks Vladimir. I've already downloaded the mozilla-central project from Alex.
I will work on it tomorrow.

There is a large memory leak in the CSS support which holds 900MB of memory when IDE is on 1.5GB of memory, when IDE reaches 2GB limits it consumes 1.5GB.
Most of the classes are:
char[], StringBuilder, FixedTextGrammarElement which in fact are single reference chain: char[]<-StringBuilder<-FixedTextGrammarElement .
These instances are held by PropertyDefinition.CACHE which is referenced by various subtypes of the CssModule registered in global lookup and hold by it.
The CACHE itself is a WeakHashMap<FileObject,GroupGrammarElement>. The WHM conditions are satisfied,  the map s used and value does not refer to key. However the map grows very fast and the FIleObjects are still alive so the values are not freed.

If you are interested in heap dump it's available.
I will attach some visual VM screens.

Created attachment 127403 [details]
Classes with most instances

Created attachment 127404 [details]
GC root path

The caching has been changed recently due to some architecture changes, but the main problem here is IMO not the cache but the size/number of the held objects. It should definitively not be such big. I'll take a more detailed look...

marian, why you changed this to P1?

The memory leak happens for me even when task list (Action Items) is closed. The CSS TLIndexer (TaskList indexers) is called even when the Task list is closed. I personally believe that it's wrong the task list indexers should not be called when the task list is closed and the indexers should be refreshed when the task list is opened. But this will require new API in task list. Probably something we should look at as it affects all project types.

fixed in web-main#552597ead66c

I had to restrict the possibilities of CssEditorModule in terms of what properties it can inject so a reasonable caching is possible.

Integrated into 'main-golden', will be available in build *201211100001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/552597ead66c
User: Marek Fukala <mfukala@netbeans.org>
Log: #221331 - "Action Items" tab makes C/C++ IDE unusable (PropertyDefinition.CACHE 1.5Gb RAM)

*** Bug 221936 has been marked as a duplicate of this bug. ***

*** Bug 221772 has been marked as a duplicate of this bug. ***

I don't know if this is related, but even the dev build is still having this problem:

http://netbeans.org/bugzilla/show_bug.cgi?id=221840

*** Bug 221772 has been marked as a duplicate of this bug. ***

*** Bug 221772 has been marked as a duplicate of this bug. ***

*** Bug 222173 has been marked as a duplicate of this bug. ***