This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 20190 - Persistent storage of Lookup.getDefault ()
Summary: Persistent storage of Lookup.getDefault ()
Alias: None
Product: platform
Classification: Unclassified
Component: Lookup (show other bugs)
Version: 3.x
Hardware: All All
: P2 blocker (vote)
Assignee: Jesse Glick
Depends on: 29831 29832 29833 29834 30494
Blocks: 17722
  Show dependency tree
Reported: 2002-02-06 09:40 UTC by Jaroslav Tulach
Modified: 2008-12-22 17:52 UTC (History)
3 users (show)

See Also:
Issue Type: TASK
Exception Reporter:

The patch of core and openide that demonstrates the 5% speedup (31.14 KB, patch)
2002-02-14 10:56 UTC, Jaroslav Tulach
Details | Diff
Current working patch (against dev sources) (25.00 KB, patch)
2002-12-24 02:59 UTC, Jesse Glick
Details | Diff
Revised working patch, which keeps a hash of relevant files and timestamps (37.35 KB, patch)
2003-01-07 04:06 UTC, Jesse Glick
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jaroslav Tulach 2002-02-06 09:40:21 UTC
Few seconds during startup are wasted in processing content of Services/
directory in spite of that it was processed in previous session. We could speed
up the startup by storing all the FolderLookup.Pairs in a persistent way and
just reload them on startup instead of initializing all the XML settings
Comment 1 Jaroslav Tulach 2002-02-06 09:42:03 UTC
First of all one has to measure what would be the possible gain, but I
expect it to be huge...
Comment 2 Jaroslav Tulach 2002-02-14 10:52:51 UTC
Measured. The implementation seems to gave 5% off the startup time -
but that still remain issue 20537 - causing few exceptions on startup
preventing the persistent state be restored correctly and thus slowing
things down - because the main point of persistent state - not to
check the files on disk any further cannot be fully achieved.

But even the partial implementation was able to get the startup time
from 20.1s to 19.1s on my computer.

Comment 3 Jaroslav Tulach 2002-02-14 10:56:03 UTC
Created attachment 4703 [details]
The patch of core and openide that demonstrates the 5% speedup
Comment 4 Jaroslav Tulach 2002-02-14 10:58:43 UTC
To use the patch start the IDE with:

 -J-Dnetbeans.lookup=quite -J-Dorg.netbeans.log.startup=print

at the end it will store the state of lookup to $USER_DIR/lookup.ser
and if you start it again it will reuse the lookup.ser and restore its
content in the NbTopManager.modulesClassPathInitialized ().
Comment 5 Jesse Glick 2002-02-19 13:02:28 UTC
What happens if the user has modified some settings on disk while the
IDE was shut down? E.g. user keeps $userdir/system/ under CVS. While
the IDE is not running, user does a cvs up and gets new settings from
other people. Or reverts some changes. Will they be visible when the
IDE is next startup or will the lookup cache need to be cleared?

Also it is not the case that adding "implements Serializable" to
various public Lookup impls is an incompatible API change if these
classes are not final? An existing subclass might be written to
contain nonserializable instance fields.
Comment 6 Jaroslav Tulach 2002-02-19 16:27:27 UTC
Ad. change when ide is shutdown: The lookup should think that there is
an instance, but when queried - it should update it state.
allInstances ().size () == 0, but allItems ().size () > 0 && allItems
().get (0).getInstance () == null so the lookup should slowly get into
the up-to-date state.

Btw. I thought that we should do some checks to find out whether the
disk has changed since last session and in such case disable the cache

Ad. Serializable - it is not too compatible, but it was just a sketch.
Comment 7 David Strupl 2002-06-25 09:39:40 UTC
Not for 3.4 - sorry.
Comment 8 Jesse Glick 2002-06-25 20:43:55 UTC
If you do implement this, please do not forget to mention any possible
caveats for module authors in the Upgrade Guide.
Comment 9 Jesse Glick 2002-09-10 17:29:04 UTC
Probably this should get reassigned to me - Trung, any opinion?
Comment 10 Jesse Glick 2002-11-12 16:12:53 UTC
Is this still planned for 4.0?
Comment 11 Jesse Glick 2002-12-23 23:12:11 UTC
I'm trying to work on this.
Comment 12 Jesse Glick 2002-12-24 02:55:42 UTC
Have an apparently working patch, but it doesn't seem to help much at all:

Average times: unoptimized  15.5721220175425
                 std. dev.  0.452610789915199
                 optimized  15.4287937323252
                 std. dev.  0.417569279980071
Improvement:                0.143328285217285
Percentage:                 0.920415888443596%

Also the patch does not yet check for cache hits; it assumes it is
always a hit. To check properly will require checking timestamps of
all module JARs (or just piggyback on the layer cache stamp?) and
adding in a stamp of $userdir/system/Services/ (i.e. customized
services), which I estimate would add at least 100msec - almost as
much as is gained here. Does not look worth the trouble (and possible
bugs) so far.
Comment 13 Jesse Glick 2002-12-24 02:59:03 UTC
Created attachment 8394 [details]
Current working patch (against dev sources)
Comment 14 Jesse Glick 2002-12-24 03:09:04 UTC
However with -nogui -nosplash the numbers are much clearer:

Average times: unoptimized  11.674832812945
                 std. dev.  0.257386849350679
                 optimized  10.7489837884903
                 std. dev.  0.0211098169661008
Improvement:                0.925849024454752
Percentage:                 7.93029792622102%
Comment 15 _ ttran 2003-01-03 10:49:42 UTC
Jesse, perhaps you should do the measurements on S1S with a bigger set
of modules to see how it scales
Comment 16 Jesse Glick 2003-01-06 18:43:36 UTC
"perhaps you should do the measurements on S1S with a bigger set of
modules to see how it scales" - good idea, thanx.
Comment 17 Jesse Glick 2003-01-07 04:06:22 UTC
Created attachment 8460 [details]
Revised working patch, which keeps a hash of relevant files and timestamps
Comment 18 Jesse Glick 2003-01-07 05:04:51 UTC
Stats w/ timestamp check on dev S1SEE build in ramdisk:

Average times: unoptimized  18.3228270292282
                 std. dev.  0.0374218846785285
                 optimized  18.0972263892492
                 std. dev.  0.0129139749485877
Improvement:                0.225600639979046
Percentage:                 1.23125454177552%

Still an improvement but not too exciting.
Comment 19 Jesse Glick 2003-01-07 05:25:13 UTC
Same test in NB (stable-with-apisupport):

Average times: unoptimized  11.8029627958934
                 std. dev.  0.189792507251049
                 optimized  10.895353770256
                 std. dev.  0.00903014360598436
Improvement:                0.90760902563731
Percentage:                 7.68967115572962%

So it seems the timestamp check is not a major factor, but something
(as yet TBD) in S1S actually slows things down - note that the
absolute savings in NB is much larger.

Perhaps lookup deser is slower in S1S for some reason - but this would
seem odd since the S1S lookup cache is 22K compared to NB's 18K,
hardly a big difference: a few extra executors.
Comment 20 Jesse Glick 2003-01-07 05:40:16 UTC
Extra time loading lookup cache in S1S: 1130ms rather than 793ms;
extra time storing: 115ms rather than 78ms. That explains only about
half the loss of improvement, even assuming that loading lookup
directly from memory (parsing *.settings etc.) is free, which is
certainly isn't.
Comment 21 Jesse Glick 2003-01-07 06:23:12 UTC
Profiling on S1SEE with lookup cache on reveals that shutdown is very

1. XMLSettingsHandler.searchFolder is rather slow, because it calls
getCookie on every .settings file, forcing them all to be parsed
though they had not been before! This could surely be optimized.

2. Many modules do stuff in close() or closing() involving system
options, forcing the options to be read for the first time, and are
really dumb about it. E.g. DatabaseOption checks all JDBC drivers!
Database module tries to store its system option, it looks like.

2a. By far the worst, however, accounting for no less than 9.7% of
*total time to start up and shut down again*, is jwd's installer,
which in close() seems to do a lot of class loading, compounded by
some apparent performance problems in ProxyClassLoader.getPackage for
deeply dependent modules (jwd appears to depend on just about
everything else).

I suspect that improvement would be much more marked if the shutdown
sequence (post-GUI hiding stuff) were excluded from timing results.
Comment 22 Jesse Glick 2003-01-07 06:30:30 UTC
"I suspect that improvement would be much more marked if the shutdown
sequence (post-GUI hiding stuff) were excluded from timing results." -
actually probably not, since ModuleSystem.shutDown must complete
successfully before WindowUtils.hideAllFrames. Probably that should be
changed: ModuleManager.closing(), if successful, should be followed by
WindowUtils.hideAllFrames, then by .close(). Most of the wasted time
in S1SEE (maybe 80%) was in close(), not closing().
Comment 23 _ ttran 2003-01-07 15:02:22 UTC
> Probably that should be changed: ModuleManager.closing(), if
> successful, should be followed by WindowUtils.hideAllFrames,
> then by .close()

Comment 24 Jesse Glick 2003-01-07 16:05:03 UTC
No, the exit code is not really part of window system, it is more tied
to the module system etc. There is nothing wrong with hideFrames, it
is simply being called a bit too early by NbTopManager.

I will commit what I have since it seems to work fine, and does
improve startup time - though shutdown time for S1SEE may be slowed
down by almost as much.
Comment 25 Jesse Glick 2003-01-07 19:12:26 UTC
committed   * Up-To-Date  1.13        core/
added       * Up-To-Date  1.1        
committed   * Up-To-Date  1.176      
committed   * Up-To-Date  1.92        openide/
committed   * Up-To-Date  1.124      
committed   * Up-To-Date  1.20       
committed   * Up-To-Date  1.126      
committed   * Up-To-Date  1.26       
committed   * Up-To-Date  1.16       
committed   * Up-To-Date  1.12       
Comment 26 Marian Mirilovic 2003-01-10 13:39:29 UTC
change lookup/core -> lookup/openide
Comment 27 Marian Mirilovic 2003-01-13 16:09:54 UTC
Note: affected source code is in core, not openide