This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
|Summary:||Cache miss in knownValidProjectXmlCRC32s during test|
|Product:||projects||Reporter:||Jaroslav Tulach <jtulach>|
|Component:||Ant Project||Assignee:||Jesse Glick <jglick>|
|Severity:||normal||CC:||jglick, jtulach, tzezula|
|Issue Type:||DEFECT||Exception Reporter:|
|Bug Depends on:|
Description Jaroslav Tulach 2011-02-03 13:46:07 UTC
Created attachment 105621 [details] Violators The whitelist test http://wiki.netbeans.org/FitnessViaWhiteAndBlackList started to fail. More than 200 of new XML related classes is being loaded. Most of them seem to be loaded by ProjectXMLCategoryReader: <violator class="org.apache.xerces.xs.datatypes.XSQName"> <stacktrace instantiator="ModuleCL@6e267b76[org.netbeans.libs.xerces]"> org.netbeans.test.ide.BlacklistedClassesHandlerSingleton.publish(BlacklistedClassesHandlerSingleton.java:369) java.util.logging.Logger.log(Logger.java:458) java.util.logging.Logger.doLog(Logger.java:480) java.util.logging.Logger.log(Logger.java:544) org.netbeans.ProxyClassLoader.loadClass(ProxyClassLoader.java:191) java.lang.ClassLoader.loadClass(ClassLoader.java:252) java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) java.lang.ClassLoader.defineClass1(Native Method) java.lang.ClassLoader.defineClass(ClassLoader.java:621) org.netbeans.JarClassLoader.doLoadClass(JarClassLoader.java:264) org.netbeans.ProxyClassLoader.selfLoadClass(ProxyClassLoader.java:300) org.netbeans.ProxyClassLoader.loadClass(ProxyClassLoader.java:227) java.lang.ClassLoader.loadClass(ClassLoader.java:252) java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) org.apache.xerces.impl.dv.xs.QNameDV.getActualValue(Unknown Source) org.apache.xerces.impl.dv.xs.XSSimpleTypeDecl.getActualValue(Unknown Source) org.apache.xerces.impl.dv.xs.XSSimpleTypeDecl.validate(Unknown Source) org.apache.xerces.impl.xs.traversers.XSAttributeChecker.checkAttributes(Unknown Source) org.apache.xerces.impl.xs.traversers.XSAttributeChecker.checkAttributes(Unknown Source) org.apache.xerces.impl.xs.traversers.XSDElementTraverser.traverseLocal(Unknown Source) org.apache.xerces.impl.xs.traversers.XSDHandler.traverseLocalElements(Unknown Source) org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source) org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source) org.apache.xerces.impl.xs.XMLSchemaLoader.loadGrammar(Unknown Source) org.apache.xerces.impl.xs.XMLSchemaLoader.loadGrammar(Unknown Source) org.apache.xerces.jaxp.validation.XMLSchemaFactory.newSchema(Unknown Source) org.netbeans.modules.project.ant.ProjectXMLCatalogReader.projectXmlCombinedSchema(ProjectXMLCatalogReader.java:189) org.netbeans.modules.project.ant.ProjectXMLCatalogReader.validate(ProjectXMLCatalogReader.java:154) org.netbeans.modules.project.ant.AntBasedProjectFactorySingleton.loadProjectXml(AntBasedProjectFactorySingleton.java:382) org.netbeans.modules.project.ant.AntBasedProjectFactorySingleton.loadProject(AntBasedProjectFactorySingleton.java:235) org.netbeans.api.project.ProjectManager.createProject(ProjectManager.java:365) org.netbeans.api.project.ProjectManager.access$300(ProjectManager.java:85) org.netbeans.api.project.ProjectManager$2.run(ProjectManager.java:286) org.netbeans.api.project.ProjectManager$2.run(ProjectManager.java:233) org.openide.util.Mutex.readAccess(Mutex.java:330) org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:233) org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:131) org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:104) org.netbeans.modules.java.project.ProjectClassPathProvider.findClassPath(ProjectClassPathProvider.java:68) org.netbeans.api.java.classpath.ClassPath.getClassPath(ClassPath.java:630) org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.findDependencies(RepositoryUpdater.java:1320) org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.access$3400(RepositoryUpdater.java:156) org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$RootsWork.getDone(RepositoryUpdater.java:2994) org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doTheWork(RepositoryUpdater.java:2159) org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task._run(RepositoryUpdater.java:3830) org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.run(RepositoryUpdater.java:3769) org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:610) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:619)
Comment 1 Jesse Glick 2011-02-03 15:15:57 UTC
When you load an Ant-based project, project.xml must be validated against schema. This requires a bunch of stuff in Xerces. Not news; true since 6.5. (As of #142680, a given project.xml is not revalidated at every startup so long as it is not modified.)
Comment 2 Jaroslav Tulach 2011-02-03 20:18:28 UTC
But now it is getting validated again! It was not validating a week ago, now it validates for some reason. Have we changed the format of the project files? If so, we need to fix the test to contain newer version...
Comment 3 Jesse Glick 2011-02-03 22:20:13 UTC
No idea what changed, but I think it is more likely a problem with the test than the code.
Comment 4 Tomas Zezula 2011-02-04 08:04:35 UTC
There are no changes in the j2se project schema. Jardo, which project type are you using?
Comment 5 Quality Engineering 2011-02-05 06:08:52 UTC
Integrated into 'main-golden', will be available in build *201102050000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/4ed84992d14f User: Jesse Glick <email@example.com> Log: Reducing log verbosity, making level=FINE more useful for diagnosis of e.g. #195029.
Comment 6 Jaroslav Tulach 2011-02-07 08:48:11 UTC
@Tomáš: Good to know there were no XSL and schema changes. That increases the likehood this is really problem in the code. I don't exactly what version the test uses, but to find out: $ cd ide.kit/test/whitelist $ cat readme.txt and typing $ ant seems to start download of the ZIP file with the project. The URL is http://hg.netbeans.org/binaries/FDCBE13B6CA7B154F6EE29BEF3B9404F7A7B1A57-lime6.zip @Jesse's changes: I don't understand why you removed REGRESSION, PERFORMANCE keywords (the behavior is changed compared to 6.9 and changed to be worse) and changed priority to P3 (in addition to the above, no changes can be propagated from ergonomics build). So this is clearly not a "test p3 & ignore" problem.
Comment 7 Tomas Zezula 2011-02-07 08:59:21 UTC
There were no XSD changes by t XSL was updated. This may be the reason. Anyway I will try.
Comment 8 Jaroslav Tulach 2011-02-07 09:52:28 UTC
I am trying to investigate too. I can start the test in $ cd ide.kit/test/whitelist $ cat readme.txt $ ant -Ddebug.port=4000 When I place breakpoint to AntBasedProjectFactorySingleton:370 (around the known hashes check), it is not invoked at all during first run (good, there is no project). The breakpoint is hit for the second time. CRC value is computed as 0xa891a89f, but knownHashes is empty. Thus the code starts to validate(!). The computed CRC is then added to the knownHashes, but possibly before the upgrade of the files(?). Then there were next two hash values added into the knownHashes preferences node asynchronously. Possibly, these may not be written to the knownHashes set at all in the test(?), if it finishes sooner. That is all I know by now.  slightly surprising to me as this means the code always validates the XML files until it accumulates the list of known hashes having some well known precomputed in newly installed build would be more effective
Comment 9 Jaroslav Tulach 2011-02-07 11:00:35 UTC
I chatted briefly with Tomáš. The CRC check is done in loadProject. The upgrade of XML files code is run in ProjectHook (e.g. later). The upgrade is handled by some helper classes in project.ant - these classes shall register the new CRC check. Passing to Jesse to deal with that. In addition Tomáš mentioned that the proper CRC values are in some properties file. The need for usages of Preferences for storing CRC values is unclear. Why it is not enough to check the value in nbproject/genfiles.properties and skip the validation if the value is the same?  This would eliminate the need to validate anything on the "first" start when the preferences node is not filled yet.
Comment 10 Jesse Glick 2011-02-07 16:04:11 UTC
(In reply to comment #6) > I don't understand why you removed REGRESSION, PERFORMANCE keywords There is no demonstrated change in behavior of the IDE, only of a test which may or may not correspond to a real use case. When I manually checked the behavior of the XML validation cache in a dev build, it worked fine. > no changes can be propagated from ergonomics build Stop running this test in the ergonomics build then. I certainly made no changes in the ergonomics repo so it makes no sense for this test to be blocking propagation of unrelated changes from that repo. (In reply to comment #8) > having some well known [hashes] > precomputed in newly installed build would be more effective Not possible since each project will typically differ at least in <name>. (In reply to comment #9) > The upgrade of XML files code is run in ProjectHook > [...] these classes shall register the new CRC check. Not sure what this means. Are you referring to org.netbeans.modules.java.j2seproject.UpdateProjectImpl? That would only be active once in the lifetime of a project, so whatever it does is not a performance issue, though the test might be inadvertently triggering this by using a very old sample project. Perhaps you meant that AntProjectHelper.saveXml could also cache the CRC of what it saves, since it has already validated that XML so it could make it slightly faster to load next time - a useful optimization, though a small benefit compared to the existing cache since it only takes effect in the case that project.xml is modified, which is much less often than it is read. > Is it not enough to check the value in nbproject/genfiles.properties and skip > the validation if the value is the same? (Assuming the project uses GeneratedFilesHelper.) This may be possible; would have the advantage of keeping the information more localized (avoiding NbPreferences), and the minor disadvantage of forcing a revalidation after making and then reverting a change to project.xml. Would also need to consider whether permitting XML validation to be bypassed by content in the project source tree itself would introduce any security risk, i.e. whether any code calling getPrimaryConfigurationData(true) could misbehave badly if given invalid data. Also need to check if this even works (without a newly introduced field specially for this purpose) if there are modifications to build.xml or build-impl.xml or if an older version of the stylesheet is present at runtime; in either case the new .data.crc32 will not be written even after a project.xml modification.
Comment 11 Jaroslav Tulach 2011-02-08 08:43:47 UTC
I think we can agree on following: (In reply to comment #10) > Perhaps you meant that AntProjectHelper.saveXml could also cache the CRC of > what it saves, since it has already validated that XML so it could make it > slightly faster to load next time - a useful optimization, OK. Based on this I added the XML validation classes to whitelist for now: ergonomics#66c7074f7c13 Revert this commit when integrating your fix. > > Is it not enough to check the value in nbproject/genfiles.properties and skip > > the validation if the value is the same? > > (Assuming the project uses GeneratedFilesHelper.) This may be possible; This would be nice in general (if the additional concerns can be solved). The test is not however going to be influenced by this additional improvement.
Comment 12 Quality Engineering 2011-02-09 06:05:47 UTC
Integrated into 'main-golden', will be available in build *201102090000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/66c7074f7c13 User: Jaroslav Tulach <firstname.lastname@example.org> Log: #195029: Temporarily allowing load of XML validation classes on startup. Caused by cache miss in the AntBasedProjectFactory.loadProject code due to upgrade of the project files to newer version not updating the cache.
Comment 13 Jesse Glick 2012-04-26 13:00:10 UTC
I see plenty of whitelist violations; reverting #66c7074f7c13 does not seem to add any, certainly nothing related to Xerces. Anyway I should be able to fix based on unit tests.
Comment 14 Jesse Glick 2012-04-26 14:02:27 UTC
Comment 15 Quality Engineering 2012-04-27 10:01:03 UTC
Integrated into 'main-golden', will be available in build *201204270400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/7fa214936dcf User: Jesse Glick <email@example.com> Log: #195029: cache miss in knownValidProjectXmlCRC32s during test. Save the checksum of project.xml being written (after it has been validated), so it would not need to be redundantly validated when loaded after restart.