This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 97215 - Huge regression of Expanding Source Packages node in Projects view - first usage
Summary: Huge regression of Expanding Source Packages node in Projects view - first usage
Alias: None
Product: projects
Classification: Unclassified
Component: Generic Infrastructure (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker (vote)
Assignee: Jesse Glick
Depends on:
Blocks: 49026
  Show dependency tree
Reported: 2007-03-05 16:38 UTC by Marian Mirilovic
Modified: 2007-07-11 14:29 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:

Graph - history of measured values - Expand Source Packages node (44.99 KB, image/png)
2007-03-05 16:41 UTC, Marian Mirilovic

Note You need to log in before you can comment on or make changes to this bug.
Description Marian Mirilovic 2007-03-05 16:38:32 UTC
Performance tests found huge regression in test case : Expand Source Packages node  
(expand Source Packages node in Projects view, testing project JEdit)

on all machines for the first usage between builds 
200703011900 & 200703041900

Expand Source Packages node - 1st usage
121 % 	110 % 	108 % 	127 % 	112 % 	103 %

( Linux FC 3/1.5.0_11, 
Solaris 9/1.5.0_11, Solaris 10/1.5.0_11,
Win 2000/1.5.0_11, Win XP/1.5.0_11, Win XP/1.6.0)

Logs from repaint manager shows the same number of repaints, the only one
difference is that all paints are taking more time :

Subsequent invocation is fine
Comment 1 Marian Mirilovic 2007-03-05 16:41:49 UTC
Created attachment 39177 [details]
Graph - history of measured values - Expand Source Packages node
Comment 2 Petr Nejedly 2007-03-06 15:17:20 UTC
Nothing has changed recently in the nodes+explorer, so It must be either some
new and really slow loader added (profiler?) or some change in DataSystems/other
We need to fix this for 6.0, but I don't have time now to analyze this.
Comment 3 _ rkubacki 2007-03-07 13:25:11 UTC
I'm going to look at this. 
Comment 4 _ rkubacki 2007-03-07 14:24:29 UTC
One note about the test: system property is no-op
after Retouche merge.
Comment 5 _ rkubacki 2007-03-07 16:38:27 UTC
I cannot reproduce this regression. Or if I can it is not bigger than 25%.
Trying with the same project. The only difference is that it is not in same
folder ignored by versioning (sharability query is visible when called from
PackageView.findNonExcludedPackages in profiler results).

My suspect was includes/excludes but this is not confirmed.
Comment 6 _ rkubacki 2007-03-08 10:37:21 UTC
I already tried to put these project into directory ignored by versioning.

Passing back to Marian for further investigation.
Comment 7 Marian Mirilovic 2007-03-08 14:30:00 UTC
Radim, I measured both builds locally :

Expand Source Packages node [1]	671 ms
Expand Source Packages node [2]	 28 ms

Expand Source Packages node [1]	2114 ms
Expand Source Packages node [2]	  25 ms

I was watching the tests and I think that in the build 200703011900 nodes are
painted(added under Source Packages node) in Explorer by group of 3-5, but in
build 200703041900 each node is painted(added under Source Packages node)
separately. Logs from Repaint Manager shows that each paint of
org.openide.explorer.view.TreeView$ExplorerTree, 0,0,1115,753
takes longer.
Comment 8 _ rkubacki 2007-03-12 16:18:49 UTC
OK, now I see the difference between two these builds both in regular usage and
in profiler. It is not that big on my notebook or my Ultra-20 desktop (it is
like 50%). Slower machine running our tests can be more affected.

The biggest diffference that I see is use of SharabilityQuery. This means that
we convert FO to file and unfortunately SQ impls like ProjectSharabilityQuery
convert this back to FO and deep in stack called from MasterURLMapper.geFOs you
can see also FU.normalizeFile. 

FileOwnerQuery is now also called more often - from
SourcesHelper$SourceRoot$Group.contains as well as from ProjectSharability query.

Another problem that I see is PackageViewChildren.refreshKeysAsync that post to
EDT request to call setKeys() so we refresh keys with each added node/package
and it is almost always followed by repaint. This is probably the same as before
integration of includes/excludes but still something what worked differently
than in past where we processed larger batches IIRC.

I'm reassigning to Jesse G. who integrated excludes/includes. 
Comment 9 Jesse Glick 2007-03-24 17:18:45 UTC
PackageViewChildren.refreshKeysAsync has been used for a few months. It makes
absolute times worse but responsiveness much better. Could probably be massaged
to batch up key refreshes with some timeout ~ 2-3sec (i.e. need to show even
single refreshes within the timeout), but would need to play with it
interactively to see the effect. Package expansion is sufficiently fast on my
computer that anything I try will probably look fine.

Regarding ProjectSharabilityQuery etc. - sounds plausible; various methods could
probably be optimized to keep a cache of recent arguments, or something like
this. (For example it is likely that FileUtil.toFile{,Object} could be made
substantially faster when called many times, by keeping a LRU cache of arguments
and near ancestors.)

My test case is a simple j2seproject with JDK 7 (Subversion checkout)
j2se/src/share/classes, running on JDK 6 -Xmx512m (although it in fact never
uses >40Mb heap). I have the Retouche cache built in advance since that is slow
to create the first time. In an uninstrumented VM, Source Packages expands on my
Tecra M5 in a few seconds, with smooth progression (new packages appear steadily
and progress bar moves along).

I am trying to profile classes from several NBM projects, since the whole point
is that it is unclear exactly what is the culprit:


Unfortunately my attempts to get profiling results have failed badly. The
profiler begins to collect results but then quickly eats up more and more memory
(in the development IDE, not the target VM), leading to OOME even with -Xmx400m,
and I eventually have to killall -9 java. Heap dumps show the devel IDE full of
~150k strings and a lot of WeakReference's.
Comment 10 Jesse Glick 2007-06-22 01:51:00 UTC
I am still unable to run the profiler (see issue #107753), so I am not going to be able to work on this unless someone
can either assist me in setting up the profiler, or tell me where the problem spots are (and verify the effectiveness of
proposed patches).
Comment 11 Jesse Glick 2007-06-27 21:02:27 UTC
See if these changes helped any.

Checking in ant/project/src/org/netbeans/spi/project/support/ant/;
/shared/data/ccvs/repository/ant/project/src/org/netbeans/spi/project/support/ant/,v  <--
new revision: 1.20; previous revision: 1.19
Checking in projects/queries/src/org/netbeans/api/queries/;
/shared/data/ccvs/repository/projects/queries/src/org/netbeans/api/queries/,v  <--
new revision: 1.8; previous revision: 1.7
Checking in java/project/src/org/netbeans/spi/java/project/support/ui/;
/shared/data/ccvs/repository/java/project/src/org/netbeans/spi/java/project/support/ui/,v  <--
new revision: 1.24; previous revision: 1.23
Checking in java/project/src/org/netbeans/spi/java/project/support/ui/;
/shared/data/ccvs/repository/java/project/src/org/netbeans/spi/java/project/support/ui/,v  <--
new revision: 1.81; previous revision: 1.80
Checking in projects/projectapi/src/org/netbeans/modules/projectapi/;
new revision: 1.16; previous revision: 1.15
Comment 12 Oleg Khokhlov 2007-07-11 14:29:40 UTC
verified with latest builds