Bug 57472

Summary: performance (classloader?) problems with signed jars in WEB-INF/lib
Product: Tomcat 8 Reporter: ole.schulz-hildebrandt
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Severity: normal    
Priority: P2    
Version: 8.0.17   
Target Milestone: ----   
Hardware: PC   
OS: Linux   

Description ole.schulz-hildebrandt 2015-01-21 08:56:05 UTC
cf. http://mail-archives.apache.org/mod_mbox/tomcat-users/201501.mbox/%3CADA7A9443C47CC4E9C065E9C84C6891D6B3994A3B7%40ex01.ppinet.de%3E

After moving one of our web applications from Tomcat 7 to Tomcat 8 (8.0.17, Java 1.8.0_25, CentOS 6) the time for deploying and initializing the webapp increased by a factor of 30 (6s vs. 180s). Analyzing the problem we found out that it had to do with a signed jar in the WEB-INF/lib of the webapp. It is a 8mb self-signed jar of the jython-library from which many classes are used in the initializing process of our webapp. When using a non-signed version of the jar the time for deploying and initializing is the same in Tomcat 8 as in Tomcat 7. 

Profiling and debugging the webapp it turned out that loading a class (and its depending classes) from the signed jar took extremely long with Tomcat 8 in comparison to Tomcat 7. 

The following is just a hypothesis of a "Tomcat source newbie": Could it be that (at least some parts of) the verification process for the whole signed jar is done again and again each time a class from a signed jar is loaded? Looking at org.apache.catalina.webresources.JarResource:47  (http://svn.apache.org/repos/asf/tomcat/tc8.0.x/trunk/java/org/apache/catalina/webresources/JarResource.java) a new instance of java.util.jar.JarFile is created each time a single class-file is read from the jar. Thus the following call of jarFile.getInputStream(jarEntry) in JarResource:50 causes a call of JarFile.initializeVerifier() (cf. JarFile:442ff in http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/jar/JarFile.java). JarFile.initializeVerifier() seems to do some precalculations (hashing?) for all files listed in the manifest of the signed jar (cf. JarFile:365ff). Thus each time a single class is loaded from a signed jar precalculations for all class files in the signed jar seem to be done. Could this be the reason for our performance problems (especially in connection with a 8mb signed jar ;-))?

If neccessary I can provide the mentioned 8mb self-signed jar of the jython-library and/or create a simple sample web application that demonstrates the issue.
Comment 1 Mark Thomas 2015-01-23 15:19:29 UTC
8MB+ will be too big for a Bugzilla attachment but if you can put the file somewhere I can download it, a web application that reproduces this issue would be much appreciated.
Comment 2 ole.schulz-hildebrandt 2015-01-23 16:51:16 UTC
I put an example application signedjar.war here: https://drive.google.com/file/d/0By0Iea2JkjtKdlI2SGFSaVg2ZWs/view?pli=1
Comment 3 Mark Thomas 2015-01-26 19:13:14 UTC
I can confirm that it is JAR verification that is causing the slow down. The way to avoid that is to keep the JAR open for the life of the web application.

However, that causes a catch-22 problem. Keeping the files open will prevent the web application from being deleted. If the web application can't be deleted folks that undeploy that way won;t be able to so the web application can't be stopped (which would be the trigger to close the file).

I need to think about the best way to fix this.
Comment 4 Christopher Schultz 2015-01-26 22:16:49 UTC
Maybe a ThreadLocal list of JARs to close?

Presumably, the ClassLoader is going to load a whole bunch of classes from these JAR files at once; probably the first time some feature/group of classes is used. When a request is completed, maybe we could tell the WebappClassLoader that the request is done and it can close any JAR files it opened during that request, rather than having it re-open the JAR each time it loads a class.
Comment 5 Mark Thomas 2015-01-26 22:26:33 UTC
That is sort of what Tomcat 7 does. I think I can port the openJars / closeJars from Tomcat 7's WebappClassLoader to Tomcat 8's resources implementation.
Comment 6 Mark Thomas 2015-01-27 19:43:07 UTC
This has been fixed in trunk and 8.0.x and will be included in 8.0.19 onwards.
Comment 7 Andrei Costescu 2015-01-28 09:05:15 UTC
Thank you! :)
Comment 8 ole.schulz-hildebrandt 2015-01-28 10:02:01 UTC
thanks for the fast fix of this issue :-)
Startup with tomcat 8.0.x trunk is as fast as tomcat 7 now.
Do I have to close this bug or is that done by you?