This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 20628

Summary: Binary XML for layers and settings
Product: platform Reporter: _ pkuzel <pkuzel>
Component: Module SystemAssignee: Jesse Glick <jglick>
Status: RESOLVED FIXED    
Severity: blocker CC: issues, jtulach, phamernik, pnejedly, rmatous
Priority: P3 Keywords: PERFORMANCE
Version: 3.x   
Hardware: PC   
OS: All   
Issue Type: ENHANCEMENT Exception Reporter:
Bug Depends on: 29354, 29356    
Bug Blocks: 20168    

Description _ pkuzel 2002-02-18 10:42:11 UTC
Modules currently contain layers in XML format. These are not intended to be
edited by a end user - no human readability is required. It causes performance
problems to parse these general XML documents during start up.

To eliminate the performance problem we can use Google's "binary xml" search
results. Particurarly compile layer's XML into its binary form. XMLFileSystem
then will sniff a magic and use faster binary reader instead of XML parser.

This approach also can eliminate bugs in layers as the compiler may do some
extra checks.

Please evaluate the idea. It is worthy for 3.4?
Comment 1 _ pkuzel 2002-02-18 10:43:56 UTC
APISupport could provide the compiler.
Comment 2 Jesse Glick 2002-02-18 14:11:23 UTC
Is it really worth the bother? According to pnejedly's studies,
parsing complete premerged layers, after removing various extra attrs
we plan to remove, takes around 300msec. If we did a lot of work to
get it down to 100msec (say) using a binary format, is that a good use
of effort (relative to more difficult debugging, more complex and
maybe buggier code, more complex build procedure for people to learn,
etc.)?

We can do validity checks much more easily in other ways, build-time
or test-time, so I don't think that's a strong argument for this.
Comment 3 Petr Nejedly 2002-02-18 14:22:00 UTC
My words. Moreover, we'd like to reduce the (startup parsed) content
of the lookup using Naming, so it is less appealing even for .settings
Reducing the priority, we may evaluate it later but it seems
as a workaround of a problem which we should solve some better way.
Comment 4 _ pkuzel 2002-02-18 14:37:25 UTC
If the parsing takes 300msec, and expected gain is about 50% then it
really does not matter.

Besides basic impl is extremely simple. You write a SAX handler that
writes down into stream callbacks it gets. Then you simply replay the
callbacks from stream instead of parsing XML document.

Thanks for the evaluation
Comment 5 Jesse Glick 2002-03-04 18:29:39 UTC
#20168's org.netbeans.core.projects.cache.LayerCacheManager would need
to be implemented and registered in place of the current XML cache.

Here is an idea: rather than using an XML format, use a JAR file to
cache the merged layers. Probably a JAR is faster to open (native
optimized code). Files can be served on demand. Attributes could be
handled by subclassing JarFileSystem and using DefaultAttributes as
its attr; so the cache recreation would store .nbattrs files. Less
data need be held in memory, e.g. all the hundreds of files added by
vcsgeneric and never used in an average IDE session would exist only
as JAR entries - assuming JarFileSystem is written correctly to not
create FileObject's until they are wanted.
Comment 6 Petr Nejedly 2002-03-18 16:07:53 UTC
AFAIK JarFS will initially create only CacheEntries, not FileObjects,
but I'm not sure it is better when it comes to memory usage...
Comment 7 Marek Grummich 2002-07-22 11:22:17 UTC
Set target milestone to TBD
Comment 8 Marek Grummich 2002-07-22 11:24:28 UTC
Set target milestone to TBD
Comment 9 rmatous 2002-07-23 12:13:04 UTC
Comparing gain of binary cache with drawbacks, seems to me that I`d
rather won`t fix this RFE.
Comment 10 Jesse Glick 2002-09-11 15:49:18 UTC
x
Comment 11 Jesse Glick 2002-09-11 15:56:05 UTC
I am reconsidering this for 4.0. I don't think the savings would be
very large compared to the XML layer cache, but there would be some,
and it may be worth the effort - especially for installations with
100+ modules, e.g. S1S EE. It could save a second or so of startup
time, I guess. It might also save a certain amount of heap, if most
files are not really accessed - consider for example the vcsgeneric
command list, hundreds of .class files in the layer which might never
be touched.

org.netbeans.core.projects.cache.* already has some infrastructure,
incl. layer parsing, cache recreation logic, and even unit tests; all
that is needed is code to write a binary file (not JAR - some custom
optimized format, possibly memory-mapped under 1.4), and code to read
it into a FileSystem (AbstractFileSystem is probably unnecessary
overhead?).
Comment 12 Jesse Glick 2002-09-11 16:02:25 UTC
Note that the cache infrastructure is entirely runtime, not
build-time: when started, NB checks for an existing cache and examines
JAR timestamps. It is recreated if JARs have been modified (or added
or deleted). The layer parser can afford to spend a little time
validating the layers, since it is not called on most startups.
(Currently it doesn't, but that could be changed.)
Comment 13 Petr Nejedly 2002-10-01 17:59:48 UTC
I've implemented a BinaryFS, a filesystem over a binary snapshot.
The FS is creating the FOs on demand (nearly-zero initialization).
I've also written a cache manager for it, but it can't work correctly
with current LCM api.

I've created a core branch named binary_layer_Oct_02 and added these
files to the CVS, you can try it and fix the LCM to allow changing
the instance of the FS at runtime (or to fall back to XFS on change)
To enable it, use
-J-Dnetbeans.cache.layers=org.netbeans.core.projects.cache.BinaryCacheManager
It won't work correctly on the first start but the next start should
be OK.
I'll be on vacations next two weeks so it is your turn now.
(BTW: The performance looks good so far)
Comment 14 Jesse Glick 2002-12-05 19:08:27 UTC
Petr next time you make a branch, please make a base tag so I can
easily work with it...
Comment 15 Jesse Glick 2002-12-05 22:29:54 UTC
Got it working. Improvement over XML cache manager:

988 msec
6.16%
1.83 standard deviations

Refactored the LayerCacheManager SPI and its callers to accommodate
both cache managers that can load from disk into an existing
filesystem (e.g. XMLLayerCacheManagerImpl) and those that cannot (e.g.
the new BinaryCacheManager).

Wrote a unit test for the new manager. Petr, it passed the automated
test without modification (only LCM semantic changes were needed).

Added a magic header to the binary cache to serve in identifying such
files.

Refactored ModuleLayeredFileSystem a bit to use a special non-caching
manager, rather than lots of if-conditions checking for null. Use of
XMLFileSystem completely factored into the non-caching manager.

Changed layer-stamp.txt format to include cache manager class name in
hash. This was necessary to permit a userdir containing an old
all-layers.xml to be used with the binary manager in effect: there is
a cache miss, the new all-layers.dat is created, then everything
proceeds normally.
Comment 16 Jesse Glick 2002-12-05 23:23:56 UTC
Petre - "utf8" is not a valid encoding under JDK 1.3, though JDK 1.4
accepts it. You must use "UTF-8", the canonical name according to the
docs.
Comment 17 Jesse Glick 2002-12-06 00:08:24 UTC
committed   * Up-To-Date  1.23       
core/src/org/netbeans/core/projects/ModuleLayeredFileSystem.java
committed   * Up-To-Date  1.2        
core/src/org/netbeans/core/projects/cache/BinaryCacheManager.java
committed   * Up-To-Date  1.2        
core/src/org/netbeans/core/projects/cache/BinaryFS.java
committed   * Up-To-Date  1.3        
core/src/org/netbeans/core/projects/cache/LayerCacheManager.java
added       * Up-To-Date  1.1        
core/src/org/netbeans/core/projects/cache/NonCacheManager.java
committed   * Up-To-Date  1.7        
core/src/org/netbeans/core/projects/cache/ParsingLayerCacheManager.java
committed   * Up-To-Date  1.4        
core/src/org/netbeans/core/projects/cache/XMLLayerCacheManagerImpl.java
added       * Up-To-Date  1.1        
core/test/unit/src/org/netbeans/core/projects/cache/BinaryCacheManagerTest.java
added       * Up-To-Date  1.1        
core/test/unit/src/org/netbeans/core/projects/cache/CacheManagerTestBaseHid.java
added       * Up-To-Date  1.1        
core/test/unit/src/org/netbeans/core/projects/cache/NonCacheManagerTest.java
committed   * Up-To-Date  1.3        
core/test/unit/src/org/netbeans/core/projects/cache/XMLLayerCacheManagerImplTest.java