Created attachment 39177 [details]
Graph - history of measured values - Expand Source Packages node

Nothing has changed recently in the nodes+explorer, so It must be either some
new and really slow loader added (profiler?) or some change in DataSystems/other
loader.
We need to fix this for 6.0, but I don't have time now to analyze this.

I'm going to look at this. 

One note about the test: perf.dont.resolve.java.badges system property is no-op
after Retouche merge.

I cannot reproduce this regression. Or if I can it is not bigger than 25%.
Trying with the same project. The only difference is that it is not in same
folder ignored by versioning (sharability query is visible when called from
PackageView.findNonExcludedPackages in profiler results).

My suspect was includes/excludes but this is not confirmed.

I already tried to put these project into directory ignored by versioning.

Passing back to Marian for further investigation.

Radim, I measured both builds locally :

200703011900
Expand Source Packages node [1]	671 ms
Expand Source Packages node [2]	 28 ms

200703041900
Expand Source Packages node [1]	2114 ms
Expand Source Packages node [2]	  25 ms

I was watching the tests and I think that in the build 200703011900 nodes are
painted(added under Source Packages node) in Explorer by group of 3-5, but in
build 200703041900 each node is painted(added under Source Packages node)
separately. Logs from Repaint Manager shows that each paint of
org.openide.explorer.view.TreeView$ExplorerTree, 0,0,1115,753
takes longer.

OK, now I see the difference between two these builds both in regular usage and
in profiler. It is not that big on my notebook or my Ultra-20 desktop (it is
like 50%). Slower machine running our tests can be more affected.

The biggest diffference that I see is use of SharabilityQuery. This means that
we convert FO to file and unfortunately SQ impls like ProjectSharabilityQuery
convert this back to FO and deep in stack called from MasterURLMapper.geFOs you
can see also FU.normalizeFile. 

FileOwnerQuery is now also called more often - from
SourcesHelper$SourceRoot$Group.contains as well as from ProjectSharability query.

Another problem that I see is PackageViewChildren.refreshKeysAsync that post to
EDT request to call setKeys() so we refresh keys with each added node/package
and it is almost always followed by repaint. This is probably the same as before
integration of includes/excludes but still something what worked differently
than in past where we processed larger batches IIRC.

I'm reassigning to Jesse G. who integrated excludes/includes. 

PackageViewChildren.refreshKeysAsync has been used for a few months. It makes
absolute times worse but responsiveness much better. Could probably be massaged
to batch up key refreshes with some timeout ~ 2-3sec (i.e. need to show even
single refreshes within the timeout), but would need to play with it
interactively to see the effect. Package expansion is sufficiently fast on my
computer that anything I try will probably look fine.

Regarding ProjectSharabilityQuery etc. - sounds plausible; various methods could
probably be optimized to keep a cache of recent arguments, or something like
this. (For example it is likely that FileUtil.toFile{,Object} could be made
substantially faster when called many times, by keeping a LRU cache of arguments
and near ancestors.)

My test case is a simple j2seproject with JDK 7 (Subversion checkout)
j2se/src/share/classes, running on JDK 6 -Xmx512m (although it in fact never
uses >40Mb heap). I have the Retouche cache built in advance since that is slow
to create the first time. In an uninstrumented VM, Source Packages expands on my
Tecra M5 in a few seconds, with smooth progression (new packages appear steadily
and progress bar moves along).

I am trying to profile classes from several NBM projects, since the whole point
is that it is unclear exactly what is the culprit:

openide{fs,masterfs}
projects/{queries,projectapi}
java/{project,j2seproject}
ant/project

Unfortunately my attempts to get profiling results have failed badly. The
profiler begins to collect results but then quickly eats up more and more memory
(in the development IDE, not the target VM), leading to OOME even with -Xmx400m,
and I eventually have to killall -9 java. Heap dumps show the devel IDE full of
~150k strings and a lot of WeakReference's.

I am still unable to run the profiler (see issue #107753), so I am not going to be able to work on this unless someone
can either assist me in setting up the profiler, or tell me where the problem spots are (and verify the effectiveness of
proposed patches).

See if these changes helped any.

Checking in ant/project/src/org/netbeans/spi/project/support/ant/SourcesHelper.java;
/shared/data/ccvs/repository/ant/project/src/org/netbeans/spi/project/support/ant/SourcesHelper.java,v  <-- 
SourcesHelper.java
new revision: 1.20; previous revision: 1.19
done
Checking in projects/queries/src/org/netbeans/api/queries/VisibilityQuery.java;
/shared/data/ccvs/repository/projects/queries/src/org/netbeans/api/queries/VisibilityQuery.java,v  <--  VisibilityQuery.java
new revision: 1.8; previous revision: 1.7
done
Checking in java/project/src/org/netbeans/spi/java/project/support/ui/PackageView.java;
/shared/data/ccvs/repository/java/project/src/org/netbeans/spi/java/project/support/ui/PackageView.java,v  <-- 
PackageView.java
new revision: 1.24; previous revision: 1.23
done
Checking in java/project/src/org/netbeans/spi/java/project/support/ui/PackageViewChildren.java;
/shared/data/ccvs/repository/java/project/src/org/netbeans/spi/java/project/support/ui/PackageViewChildren.java,v  <-- 
PackageViewChildren.java
new revision: 1.81; previous revision: 1.80
done
Checking in projects/projectapi/src/org/netbeans/modules/projectapi/SimpleFileOwnerQueryImplementation.java;
/shared/data/ccvs/repository/projects/projectapi/src/org/netbeans/modules/projectapi/SimpleFileOwnerQueryImplementation.java,v
 <--  SimpleFileOwnerQueryImplementation.java
new revision: 1.16; previous revision: 1.15
done

verified with latest builds