This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 26786

Summary: Faster parsing of module config
Product: platform Reporter: Jesse Glick <jglick>
Component: Module SystemAssignee: Jesse Glick <jglick>
Status: RESOLVED FIXED    
Severity: blocker CC: pnejedly
Priority: P2 Keywords: PERFORMANCE
Version: 3.x   
Hardware: All   
OS: All   
Issue Type: TASK Exception Reporter:
Bug Depends on: 27293, 28755    
Bug Blocks: 21675    
Attachments: Patch being tested

Description Jesse Glick 2002-08-26 16:33:09 UTC
Currently at startup:

1. Everything in Modules/*.xml must be parsed
using a plain SAX parser and the information loaded.

2. All JARs opened to check their manifests.

3. Dependencies etc. calculated.

Step #3 is probably fast enough as it is (TBD).

#1 could easily be improved: 99% of the time, the
XML files are in a well-known byte pattern and
could be "parsed" without using an XML parser -
fall back to using a real parser if they were e.g.
hand-edited.

#2 is more complicated - would require keeping
some kind of cache of module manifests separate
from the JARs themselves. Issue #21675's proposal
for a general cache of module JAR contents would
be one solution, but there might be simpler ways.
Comment 1 Jesse Glick 2002-09-03 15:39:28 UTC
Copied from #21675 (thanks for info Petr!):

---%<---
I've measured starting with 1000 dumb modules (empty layer,
no ModuleInstall, no manifest sections).

The additional time spent in startup (~11s) was caused by:
- 8s ModuleList.readInitial
    - ~4s spent in parsing separate Modules/modname.xml files
    - ~4s spent in opening the jars needed for reading manifests.
- 3s ModuleManager.enable/module preparation
    - probably mostly spent in creating the classloaders/reopening
      the module jars
---%<---

Will need to additionally check whether inter-module dependencies
consume much time (i.e. finding dependencies, calculating install
order, etc.) if there are a lot of dependencies or there are deeply
nested dependencies. I know the current partial ordering algorithms
are extremely dumb. I finally have copies of proper algorithm books at
my house so I can improve them. :-)
Comment 2 Jesse Glick 2002-09-09 23:23:33 UTC
Updates:

My measurements roughly agree with Petr's. I get an initial 7ms per
module, more or less, to parse Modules/, open JARs to check their
manifests, and so construct the initial module list.

Can take the time down to around 4ms by using a hand-optimized
"parser" and falling back to the XML parser only if that fails.

I also see around 3ms overhead for computing dependencies, bringing up
the classloader (meaning opening the JAR a second time), and skimming
over nonexistent module installers and sections. Under 5k per module
heap overhead.

Parsing XML layers is more - assuming caching is on, adding a
mid-sized layer to a module adds about 5ms to startup. It also adds a
bunch of memory: say 10-15k heap.

Opening a medium-sized JAR rather than a small one adds a little
overhead; maybe a millisecond or two, plus say 1-2k heap.
Comment 3 Jesse Glick 2002-09-10 01:33:18 UTC
Committed improved parser:

committed     Up-To-Date  1.35       
core/src/org/netbeans/core/modules/ModuleList.java
Comment 4 Jesse Glick 2002-09-11 23:03:29 UTC
I tried caching manifests. Hardly seemed to help - 95ms improvement,
less than one standard deviation. Will investigate with OptimizeIt! to
see what is going on.
Comment 5 Jesse Glick 2002-09-11 23:13:08 UTC
Created attachment 7388 [details]
Patch being tested
Comment 6 Jesse Glick 2002-09-12 22:18:21 UTC
I did some more precise testing using ModuleInitTest. It seems the
improvement is real, but not dramatic. If you have module JARs with
some interdependencies containing about 200 files apiece, and empty
layers, the per-module startup time overhead is about 6.86ms without
caching, vs. 6.32ms with caching. So, a half-millisecond speedup per
module.
Comment 7 Jesse Glick 2002-09-12 23:22:45 UTC
committed   * Up-To-Date  1.37       
core/src/org/netbeans/core/modules/Module.java
committed   * Up-To-Date  1.6        
core/src/org/netbeans/core/modules/ModuleInstaller.java
committed   * Up-To-Date  1.42       
core/src/org/netbeans/core/modules/ModuleManager.java
committed   * Up-To-Date  1.47       
core/src/org/netbeans/core/modules/NbInstaller.java
Comment 8 Petr Nejedly 2002-09-16 14:55:02 UTC
Probably the second jar opening was faster and it is slower now
(because it became first-time opening).
Comment 9 Jesse Glick 2002-09-16 16:19:06 UTC
"Probably the second jar opening was faster and it is slower now
(because it became first-time opening)." - yes, probably so. May still
be helpful for an installation with a large number of disabled
modules, though.