This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
|Summary:||Binary XML for layers and settings|
|Product:||platform||Reporter:||_ pkuzel <pkuzel>|
|Component:||Module System||Assignee:||Jesse Glick <jglick>|
|Severity:||blocker||CC:||issues, jtulach, phamernik, pnejedly, rmatous|
|Issue Type:||ENHANCEMENT||Exception Reporter:|
|Bug Depends on:||29354, 29356|
Description _ pkuzel 2002-02-18 10:42:11 UTC
Modules currently contain layers in XML format. These are not intended to be edited by a end user - no human readability is required. It causes performance problems to parse these general XML documents during start up. To eliminate the performance problem we can use Google's "binary xml" search results. Particurarly compile layer's XML into its binary form. XMLFileSystem then will sniff a magic and use faster binary reader instead of XML parser. This approach also can eliminate bugs in layers as the compiler may do some extra checks. Please evaluate the idea. It is worthy for 3.4?
Comment 1 _ pkuzel 2002-02-18 10:43:56 UTC
APISupport could provide the compiler.
Comment 2 Jesse Glick 2002-02-18 14:11:23 UTC
Is it really worth the bother? According to pnejedly's studies, parsing complete premerged layers, after removing various extra attrs we plan to remove, takes around 300msec. If we did a lot of work to get it down to 100msec (say) using a binary format, is that a good use of effort (relative to more difficult debugging, more complex and maybe buggier code, more complex build procedure for people to learn, etc.)? We can do validity checks much more easily in other ways, build-time or test-time, so I don't think that's a strong argument for this.
Comment 3 Petr Nejedly 2002-02-18 14:22:00 UTC
My words. Moreover, we'd like to reduce the (startup parsed) content of the lookup using Naming, so it is less appealing even for .settings Reducing the priority, we may evaluate it later but it seems as a workaround of a problem which we should solve some better way.
Comment 4 _ pkuzel 2002-02-18 14:37:25 UTC
If the parsing takes 300msec, and expected gain is about 50% then it really does not matter. Besides basic impl is extremely simple. You write a SAX handler that writes down into stream callbacks it gets. Then you simply replay the callbacks from stream instead of parsing XML document. Thanks for the evaluation
Comment 5 Jesse Glick 2002-03-04 18:29:39 UTC
#20168's org.netbeans.core.projects.cache.LayerCacheManager would need to be implemented and registered in place of the current XML cache. Here is an idea: rather than using an XML format, use a JAR file to cache the merged layers. Probably a JAR is faster to open (native optimized code). Files can be served on demand. Attributes could be handled by subclassing JarFileSystem and using DefaultAttributes as its attr; so the cache recreation would store .nbattrs files. Less data need be held in memory, e.g. all the hundreds of files added by vcsgeneric and never used in an average IDE session would exist only as JAR entries - assuming JarFileSystem is written correctly to not create FileObject's until they are wanted.
Comment 6 Petr Nejedly 2002-03-18 16:07:53 UTC
AFAIK JarFS will initially create only CacheEntries, not FileObjects, but I'm not sure it is better when it comes to memory usage...
Comment 7 Marek Grummich 2002-07-22 11:22:17 UTC
Set target milestone to TBD
Comment 8 Marek Grummich 2002-07-22 11:24:28 UTC
Set target milestone to TBD
Comment 9 rmatous 2002-07-23 12:13:04 UTC
Comparing gain of binary cache with drawbacks, seems to me that I`d rather won`t fix this RFE.
Comment 10 Jesse Glick 2002-09-11 15:49:18 UTC
Comment 11 Jesse Glick 2002-09-11 15:56:05 UTC
I am reconsidering this for 4.0. I don't think the savings would be very large compared to the XML layer cache, but there would be some, and it may be worth the effort - especially for installations with 100+ modules, e.g. S1S EE. It could save a second or so of startup time, I guess. It might also save a certain amount of heap, if most files are not really accessed - consider for example the vcsgeneric command list, hundreds of .class files in the layer which might never be touched. org.netbeans.core.projects.cache.* already has some infrastructure, incl. layer parsing, cache recreation logic, and even unit tests; all that is needed is code to write a binary file (not JAR - some custom optimized format, possibly memory-mapped under 1.4), and code to read it into a FileSystem (AbstractFileSystem is probably unnecessary overhead?).
Comment 12 Jesse Glick 2002-09-11 16:02:25 UTC
Note that the cache infrastructure is entirely runtime, not build-time: when started, NB checks for an existing cache and examines JAR timestamps. It is recreated if JARs have been modified (or added or deleted). The layer parser can afford to spend a little time validating the layers, since it is not called on most startups. (Currently it doesn't, but that could be changed.)
Comment 13 Petr Nejedly 2002-10-01 17:59:48 UTC
I've implemented a BinaryFS, a filesystem over a binary snapshot. The FS is creating the FOs on demand (nearly-zero initialization). I've also written a cache manager for it, but it can't work correctly with current LCM api. I've created a core branch named binary_layer_Oct_02 and added these files to the CVS, you can try it and fix the LCM to allow changing the instance of the FS at runtime (or to fall back to XFS on change) To enable it, use -J-Dnetbeans.cache.layers=org.netbeans.core.projects.cache.BinaryCacheManager It won't work correctly on the first start but the next start should be OK. I'll be on vacations next two weeks so it is your turn now. (BTW: The performance looks good so far)
Comment 14 Jesse Glick 2002-12-05 19:08:27 UTC
Petr next time you make a branch, please make a base tag so I can easily work with it...
Comment 15 Jesse Glick 2002-12-05 22:29:54 UTC
Got it working. Improvement over XML cache manager: 988 msec 6.16% 1.83 standard deviations Refactored the LayerCacheManager SPI and its callers to accommodate both cache managers that can load from disk into an existing filesystem (e.g. XMLLayerCacheManagerImpl) and those that cannot (e.g. the new BinaryCacheManager). Wrote a unit test for the new manager. Petr, it passed the automated test without modification (only LCM semantic changes were needed). Added a magic header to the binary cache to serve in identifying such files. Refactored ModuleLayeredFileSystem a bit to use a special non-caching manager, rather than lots of if-conditions checking for null. Use of XMLFileSystem completely factored into the non-caching manager. Changed layer-stamp.txt format to include cache manager class name in hash. This was necessary to permit a userdir containing an old all-layers.xml to be used with the binary manager in effect: there is a cache miss, the new all-layers.dat is created, then everything proceeds normally.
Comment 16 Jesse Glick 2002-12-05 23:23:56 UTC
Petre - "utf8" is not a valid encoding under JDK 1.3, though JDK 1.4 accepts it. You must use "UTF-8", the canonical name according to the docs.
Comment 17 Jesse Glick 2002-12-06 00:08:24 UTC
committed * Up-To-Date 1.23 core/src/org/netbeans/core/projects/ModuleLayeredFileSystem.java committed * Up-To-Date 1.2 core/src/org/netbeans/core/projects/cache/BinaryCacheManager.java committed * Up-To-Date 1.2 core/src/org/netbeans/core/projects/cache/BinaryFS.java committed * Up-To-Date 1.3 core/src/org/netbeans/core/projects/cache/LayerCacheManager.java added * Up-To-Date 1.1 core/src/org/netbeans/core/projects/cache/NonCacheManager.java committed * Up-To-Date 1.7 core/src/org/netbeans/core/projects/cache/ParsingLayerCacheManager.java committed * Up-To-Date 1.4 core/src/org/netbeans/core/projects/cache/XMLLayerCacheManagerImpl.java added * Up-To-Date 1.1 core/test/unit/src/org/netbeans/core/projects/cache/BinaryCacheManagerTest.java added * Up-To-Date 1.1 core/test/unit/src/org/netbeans/core/projects/cache/CacheManagerTestBaseHid.java added * Up-To-Date 1.1 core/test/unit/src/org/netbeans/core/projects/cache/NonCacheManagerTest.java committed * Up-To-Date 1.3 core/test/unit/src/org/netbeans/core/projects/cache/XMLLayerCacheManagerImplTest.java