This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 21675

Summary: Scalability in number of initialized modules
Product: platform Reporter: Jaroslav Tulach <jtulach>
Component: Module SystemAssignee: _ ttran <ttran>
Status: RESOLVED INVALID    
Severity: blocker CC: jglick, pnejedly, pzajac, ttran
Priority: P2 Keywords: PERFORMANCE
Version: 3.x   
Hardware: PC   
OS: Linux   
Issue Type: TASK Exception Reporter:
Bug Depends on: 21676, 22322, 26786, 27190, 29834, 31008    
Bug Blocks: 17722    
Attachments: Unfinished document

Description Jaroslav Tulach 2002-03-18 17:38:22 UTC
Find out and design solutions to enable the system to scale correctly when a lot
of modules is installed. This task should minimize the negative effect of number
of modules on startup time, memory usage as well as ensuring that the user pays
just for what he is using.
Comment 1 Petr Nejedly 2002-06-19 11:30:44 UTC
I've measured starting with 1000 dumb modules (empty layer,
no ModuleInstall, no manifest sections).

The additional time spent in startup (~11s) was caused by:
- 8s ModuleList.readInitial
    - ~4s spent in parsing separate Modules/modname.xml files
    - ~4s spent in opening the jars needed for reading manifests.
- 3s ModuleManager.enable/module preparation
    - probably mostly spent in creating the classloaders/reopening
      the module jars

Comment 2 Jesse Glick 2002-06-19 17:10:15 UTC
Parsing the Modules/*.xml files could probably be made a lot faster by
optimistically assuming that they are in a simple fixed format and
looking for particular substrings. If the assumption fails, revert to
an actual XML parse.

We may be able to create a cache of module manifests so JARs do not
need to be opened until the module is actually turned on. This could
save another chunk of time, perhaps. The basic timestamp
infrastructure already used for layer caching could help.
Comment 3 Petr Nejedly 2002-06-20 08:18:20 UTC
My guess is that the problem with Modules/* is not in XMP parsing
but in number of files. Cached layer (360KB) is parsed below a second
on my machine, while 1000x560B takes 4s while parsing itself
could be around 1.5s

It may be possible to copy-out the manifests to the module stati
and/or merge the stati to a single file (maybe even binary),
but this collides with moduleset changes on project switch.

(on the other hand, changing moduleset on project switch
quite slows down the project switch so it is a question whether
it is really that useful)
Comment 4 Petr Nejedly 2002-06-20 08:19:33 UTC
BTW: Computing the fingerprint for a XMLFS of 1000 modules
takes about 600ms itself.
Comment 5 Jesse Glick 2002-06-20 15:23:49 UTC
So you're saying just accessing all the files in Modules/*.xml is
itself a problem (LocalFileSystem + MultiFileSystem overhead on top of
raw OS speed for finding files)? Or that opening the JARs to get their
manifests is the problem? I'm confused.

Merging stati to a single file would indeed change the semantics of
the module system (not just because of project switching) so I would
like to avoid it unless it is really shown to be necessary. But
caching more manifest information in the status XML files would be
fine, I think - probably just impl version and dependencies incl.
provides-requires would need to be added. This could permit the JARs
of enabled modules to be opened only once under normal conditions, and
those of disabled modules not at all.
Comment 6 Petr Nejedly 2002-06-20 16:46:03 UTC
I'm saying that 4s is spent in opening + parsing Modules/* files.
I don't have exact distribution numbers between opening+parsing
but I extrapolated the parsing time to be about 1.5s and from
that I infered that opening takes 2.5s
1st time opening of jars takes 4s, the second opening seems
to be faster so skipping the 1st opening may not be a win itself.
I have to make more experiments yet.
Comment 7 Jesse Glick 2002-06-20 17:13:47 UTC
Created attachment 6355 [details]
Unfinished document
Comment 8 Jesse Glick 2002-06-20 17:16:05 UTC
Attaching some thoughts I had a few months ago on caching stuff during
startup, which never got finished. Note that the layer cache is
already implemented. Treat it as a collection of ideas rather than a
concrete proposal, because I know there are problems with it as written.
Comment 9 Marek Grummich 2002-07-22 11:28:52 UTC
Set target milestone to TBD
Comment 10 Marek Grummich 2002-07-22 11:30:47 UTC
Set target milestone to TBD
Comment 11 Jaroslav Tulach 2002-09-09 17:02:15 UTC
Seems more or less work for the performance team.
Comment 12 Jesse Glick 2008-07-07 15:17:20 UTC
Obsolete now.