When starting Tomcat 7.0.29 and 7.0.30, and automatically deploying a war file from webapps the following error is thrown: 14:01:06,380 ERROR [HostConfig:576] Error waiting for multi-thread deployment of context descriptors to complete java.util.concurrent.ExecutionException: java.lang.StackOverflowError at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:574) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:470) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1413) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:313) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1140) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:785) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.StackOverflowError at java.util.HashSet.(HashSet.java:103) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2243) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2260) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2260) ...(and many more identical lines)... However this works fine in 7.0.28.
Can you provide the smallest possible WAR that reproduces this issue please?
I've uploaded the minimal war file causing the issue on http://www.rent-a-car-cluj.com/app.war, since the upload on Apache is limited to 1Mb. The app in the war is stripped from JSP's, classes and any item that was not relevant for the issue. I guess the issue is related to the high number of jars/classes in the libraries. (After removing several libraries, deployment went succesfully. I could not make a correlation with any specific library. System used: Win 7/64 bits.
Perfect. I'm in the middle of something else right now but this is next on my to do list (unless someone beats me to it). The test WAR is ~35MB for folks considering downloading it.
To quote from the 7.0.29 changelog: <quote> As per section 1.6.2 of the Servlet 3.0 specification and clarification from the Servlet Expert Group, the servlet specification version declared in web.xml no longer controls if Tomcat scans for annotations. Annotation scanning is now always performed - regardless of the version declared in web.xml - unless metadata complete is set to true. </quote> I suspect this is why you see the issue with 7.0.29 onwards. However, I am unable to reproduce this, even if I drop -Xss to the bare minimum required while allowing the JVM to start. Please do not re-open this issue unless you can provide the exact steps to reproduce on a clean install of the latest JVM, Tomcat 7.0.x etc.
I had the same problem. Steps to reproduce are: Due to dependency management problems I deployed a WAR file containing two JAR files containing different versions of a library, where version 1 contained a class A which inherited from B and another version 2 which contained a class B inheriting from A. The classes were loaded in such an order that A and B cyclically inherited from each other which leads to a stack overflow in populateSCIsForCacheEntry because it does not detect cycles in the inheritance tree.
Changing to an enhancement to see what, if anything, can be done to provide a better error message.
Better error message added to trunk and 7.0.x and will be included in 7.0.34 onwards.
I'm having the same problem, but it is not constant, some times it happens, some time dont. But something that I observed was that it happens more frequently when I'm using x86_64 oracle jvm. java version "1.6.0_38" Java(TM) SE Runtime Environment (build 1.6.0_38-b05) Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode) CATALINA_OPTS="-XX:MaxPermSize=256m -Xmx1g"
This was a serious show stopper. Calling it an enhancement to make it work is confusing. It only happened with certain wars, and oddly enough I could tweak the deploypath to cause it to occur or go away. E.g. Whith one of my wars, when it was deployed to a path that started C:\Progs\... it failed with the populateSCIsForCacheEntry errors. When I changed it to C:\Prog\..., it didn't. Unfortunately, this is not a practical solution for my current situation. So until it is fixed, we are stuck at version 7.0.28. Although this issue says it has been resolved, there is no indication of how it was resolved. Did someone actually find and fix the problem, or just change the error message as suggested in by Mark Thomas? Thanks.
(In reply to comment #9) 1. There is no evidence that you have the same problem. 2. There is a number of ways to configure annotation scanning, or avoid it altogether in a specific application. The bonus point is that your application will start up faster. Bugzilla is not a support forum to go into details. It is also possible to configure a different value for -Xss. 3. We are volunteers here. Feel free to propose a better patch than r1416502 Sometimes it is possible to change algorithm and unroll a recursion. I have not dug whether it is possible here.
(In reply to comment #10) Thanks for your reply, and thanks for all the volunteer efforts on this project. It is appreciated, and I didn't mean to imply otherwise. I do believe I have the same problem. When starting up 7.0.29 and deploying two WARs, one of my WARs fails to deploy with a very similar stack trace. (I'm in deployWARs instead of deployDescriptors, but I think that must be due to a different version than the original post. Other than that and the HashMap vs. HashSet, the stacks are pretty much the same.) My stack ends with the following: ... Caused by: java.lang.StackOverflowError at java.util.HashMap.get(HashMap.java:300) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2215) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2227) at ... There are 1024 recursions at line 2227 before the stack is exhausted. I'm not able to set -Xss large enough to make any difference. The populateSCIsForCacheEntry always recurses 1024 times and throws the StackOverflowError Error. That seems like a very deep recursion. The fact that different deploy paths (<Host appBase=...) can change the behavior doesn't make sense either. It isn't fixed for me, but I will try to find another resolution. Thanks again.
The point here is that if you ship an application that does this: <quote> Due to dependency management problems I deployed a WAR file containing two JAR files containing different versions of a library, where version 1 contained a class A which inherited from B and another version 2 which contained a class B inheriting from A. The classes were loaded in such an order that A and B cyclically inherited from each other which leads to a stack overflow in populateSCIsForCacheEntry because it does not detect cycles in the inheritance tree. </quote> Then the application is going to fall over as soon as it hits those classes. That is an application bug, not a Tomcat bug. You might be lucky and not use those classes at all in your application. In which case you wouldn't see this until Tomcat was updated to scan classes for annotations as intended by the Servlet 3.0 EG. Hence it might look like a Tomcat problem. If you have a valid set of JARs that can trigger a stack overflow then that would be a new bug and one that would get fairly prompt attention.
I have the same problem in 7.0.35 version. My application is very big - over 200 jars, and it's hard to identify the jar which couse the problem. There is no information about classes in infinite loop call stack. My call stack: Caused by: java.lang.IllegalStateException: Unable to complete the scan for annotations for web application []. Possible root causes include a too low setting for -Xss and illegal cyclic inheritance dependencies at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2109) at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1981) at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1947) at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1932) at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1326) at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5179) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) ... 11 more Caused by: java.lang.StackOverflowError at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269)
I'll see what I can do to add some class and/or JAR information to the error message.
I have added the class hierarchy to the error message in trunk and 7.0.x. It will be included in 7.0.38 onwards.
I've tried to upgrade to 7.0.37 (latest release at this moment). We still encounter the same issue, but this time the logged stacktrace is more detailed than the one I've added in the initial description: 10:49:06,990 ERROR [ContainerBase:903] ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/dossier]] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:657) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1637) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.IllegalStateException: Unable to complete the scan for annotations for web application [/dossier]. Possible root causes include a too low setting for -Xss and illegal cyclic inheritance dependencies at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2109) at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1981) at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1947) at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1932) at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1326) at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5179) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) ... 11 more Caused by: java.lang.StackOverflowError at java.util.Random.nextInt(Random.java:239) at sun.misc.Hashing.randomHashSeed(Hashing.java:254) at java.util.HashMap.<init>(HashMap.java:255) at java.util.HashMap.<init>(HashMap.java:305) at java.util.HashSet.<init>(HashSet.java:103) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2252) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269) ... I thought this might be helpful for further progress on this issue. As mentioned in comment 10 and also on http://collab.sakaiproject.org/pipermail/sakai-dev/2012-November/019574.html , I was able to workaround this exception by skipping the jar scannining in catalina.properties: # Additional JARs (over and above the default JARs listed above) to skip when # scanning for Servlet 3.0 pluggability features. These features include web # fragments, annotations, SCIs and classes that match @HandlesTypes. The list # must be a comma separated list of JAR file names. org.apache.catalina.startup.ContextConfig.jarsToSkip=*.jar IMHO this is still a workaround and not the fix.
Thanks for the new class hierarchy message. Running tomcat master with that improvement showed me exactly what the problem was - a loop caused by an incompatible version of Jaxen bundled with an old dom4j. Upgrading the dependencies in my webapp fixed it.
FYI seems kind of weird but we have experienced this when for various reasons we needed to kill -9 the tomcat process. Doesn't necessarily make sense but was fairly consistent.