This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 26786 - Faster parsing of module config
Summary: Faster parsing of module config
Alias: None
Product: platform
Classification: Unclassified
Component: Module System (show other bugs)
Version: 3.x
Hardware: All All
: P2 blocker (vote)
Assignee: Jesse Glick
Depends on: 27293 28755
Blocks: 21675
  Show dependency tree
Reported: 2002-08-26 16:33 UTC by Jesse Glick
Modified: 2008-12-23 08:36 UTC (History)
1 user (show)

See Also:
Issue Type: TASK
Exception Reporter:

Patch being tested (14.60 KB, patch)
2002-09-11 23:13 UTC, Jesse Glick
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2002-08-26 16:33:09 UTC
Currently at startup:

1. Everything in Modules/*.xml must be parsed
using a plain SAX parser and the information loaded.

2. All JARs opened to check their manifests.

3. Dependencies etc. calculated.

Step #3 is probably fast enough as it is (TBD).

#1 could easily be improved: 99% of the time, the
XML files are in a well-known byte pattern and
could be "parsed" without using an XML parser -
fall back to using a real parser if they were e.g.

#2 is more complicated - would require keeping
some kind of cache of module manifests separate
from the JARs themselves. Issue #21675's proposal
for a general cache of module JAR contents would
be one solution, but there might be simpler ways.
Comment 1 Jesse Glick 2002-09-03 15:39:28 UTC
Copied from #21675 (thanks for info Petr!):

I've measured starting with 1000 dumb modules (empty layer,
no ModuleInstall, no manifest sections).

The additional time spent in startup (~11s) was caused by:
- 8s ModuleList.readInitial
    - ~4s spent in parsing separate Modules/modname.xml files
    - ~4s spent in opening the jars needed for reading manifests.
- 3s ModuleManager.enable/module preparation
    - probably mostly spent in creating the classloaders/reopening
      the module jars

Will need to additionally check whether inter-module dependencies
consume much time (i.e. finding dependencies, calculating install
order, etc.) if there are a lot of dependencies or there are deeply
nested dependencies. I know the current partial ordering algorithms
are extremely dumb. I finally have copies of proper algorithm books at
my house so I can improve them. :-)
Comment 2 Jesse Glick 2002-09-09 23:23:33 UTC

My measurements roughly agree with Petr's. I get an initial 7ms per
module, more or less, to parse Modules/, open JARs to check their
manifests, and so construct the initial module list.

Can take the time down to around 4ms by using a hand-optimized
"parser" and falling back to the XML parser only if that fails.

I also see around 3ms overhead for computing dependencies, bringing up
the classloader (meaning opening the JAR a second time), and skimming
over nonexistent module installers and sections. Under 5k per module
heap overhead.

Parsing XML layers is more - assuming caching is on, adding a
mid-sized layer to a module adds about 5ms to startup. It also adds a
bunch of memory: say 10-15k heap.

Opening a medium-sized JAR rather than a small one adds a little
overhead; maybe a millisecond or two, plus say 1-2k heap.
Comment 3 Jesse Glick 2002-09-10 01:33:18 UTC
Committed improved parser:

committed     Up-To-Date  1.35       
Comment 4 Jesse Glick 2002-09-11 23:03:29 UTC
I tried caching manifests. Hardly seemed to help - 95ms improvement,
less than one standard deviation. Will investigate with OptimizeIt! to
see what is going on.
Comment 5 Jesse Glick 2002-09-11 23:13:08 UTC
Created attachment 7388 [details]
Patch being tested
Comment 6 Jesse Glick 2002-09-12 22:18:21 UTC
I did some more precise testing using ModuleInitTest. It seems the
improvement is real, but not dramatic. If you have module JARs with
some interdependencies containing about 200 files apiece, and empty
layers, the per-module startup time overhead is about 6.86ms without
caching, vs. 6.32ms with caching. So, a half-millisecond speedup per
Comment 7 Jesse Glick 2002-09-12 23:22:45 UTC
committed   * Up-To-Date  1.37       
committed   * Up-To-Date  1.6        
committed   * Up-To-Date  1.42       
committed   * Up-To-Date  1.47       
Comment 8 Petr Nejedly 2002-09-16 14:55:02 UTC
Probably the second jar opening was faster and it is slower now
(because it became first-time opening).
Comment 9 Jesse Glick 2002-09-16 16:19:06 UTC
"Probably the second jar opening was faster and it is slower now
(because it became first-time opening)." - yes, probably so. May still
be helpful for an installation with a large number of disabled
modules, though.