Bug 53871 - java.lang.StackOverflowError on deploying a web application
Summary: java.lang.StackOverflowError on deploying a web application
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 7.0.35
Hardware: PC All
: P2 enhancement (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-09-14 12:40 UTC by Robert Pasztor
Modified: 2013-11-18 21:22 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Pasztor 2012-09-14 12:40:44 UTC
When starting Tomcat 7.0.29 and 7.0.30, and automatically deploying a war file from webapps the following error is thrown:
14:01:06,380 ERROR [HostConfig:576] Error waiting for multi-thread deployment of context descriptors to complete java.util.concurrent.ExecutionException: java.lang.StackOverflowError
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
    at java.util.concurrent.FutureTask.get(FutureTask.java:111)
    at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:574)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:470)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1413)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:313)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
    at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401)
    at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1140)
    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:785)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Caused by: java.lang.StackOverflowError at java.util.HashSet.(HashSet.java:103) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2243) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2260) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2260)
...(and many more identical lines)...

However this works fine in 7.0.28.
Comment 1 Mark Thomas 2012-09-16 18:21:05 UTC
Can you provide the smallest possible WAR that reproduces this issue please?
Comment 2 Robert Pasztor 2012-09-16 21:24:17 UTC
I've uploaded the minimal war file causing the issue on http://www.rent-a-car-cluj.com/app.war, since the upload on Apache is limited to 1Mb. 

The app in the war is stripped from JSP's, classes and any item that was not relevant for the issue.

I guess the issue is related to the high number of jars/classes in the libraries. (After removing several libraries, deployment went succesfully. I could not make a correlation with any specific library.

System used: Win 7/64 bits.
Comment 3 Mark Thomas 2012-09-16 21:39:06 UTC
Perfect. I'm in the middle of something else right now but this is next on my to do list (unless someone beats me to it).

The test WAR is ~35MB for folks considering downloading it.
Comment 4 Mark Thomas 2012-09-16 22:29:16 UTC
To quote from the 7.0.29 changelog:
<quote>
As per section 1.6.2 of the Servlet 3.0 specification and clarification from the Servlet Expert Group, the servlet specification version declared in web.xml no longer controls if Tomcat scans for annotations. Annotation scanning is now always performed - regardless of the version declared in web.xml - unless metadata complete is set to true.
</quote>

I suspect this is why you see the issue with 7.0.29 onwards.

However, I am unable to reproduce this, even if I drop -Xss to the bare minimum required while allowing the JVM to start.

Please do not re-open this issue unless you can provide the exact steps to reproduce on a clean install of the latest JVM, Tomcat 7.0.x etc.
Comment 5 Matthias Schuetz 2012-12-03 11:49:22 UTC
I had the same problem. Steps to reproduce are:

Due to dependency management problems I deployed a WAR file containing two JAR files containing different versions of a library, where version 1 contained a class A which inherited from B and another version 2 which contained a class B inheriting from A.
The classes were loaded in such an order that A and B cyclically inherited from each other which leads to a stack overflow in populateSCIsForCacheEntry because it does not detect cycles in the inheritance tree.
Comment 6 Mark Thomas 2012-12-03 12:35:02 UTC
Changing to an enhancement to see what, if anything, can be done to provide a better error message.
Comment 7 Mark Thomas 2012-12-03 13:22:33 UTC
Better error message added to trunk and 7.0.x and will be included in 7.0.34 onwards.
Comment 8 Fernando Kasten Peinado 2013-01-18 18:24:57 UTC
I'm having the same problem, but it is not constant, some times it happens, some time dont. But something that I observed was that it happens more frequently when I'm using x86_64 oracle jvm.

java version "1.6.0_38"
Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)

CATALINA_OPTS="-XX:MaxPermSize=256m -Xmx1g"
Comment 9 John Bize 2013-02-01 12:49:27 UTC
This was a serious show stopper.  Calling it an enhancement to make it work is confusing.

It only happened with certain wars, and oddly enough I could tweak the deploypath to cause it to occur or go away.  E.g. Whith one of my wars, when it was deployed to a path that started C:\Progs\... it failed with the populateSCIsForCacheEntry errors.  When I changed it to C:\Prog\..., it didn't.

Unfortunately, this is not a practical solution for my current situation.  So until it is fixed, we are stuck at version 7.0.28.  Although this issue says it has been resolved, there is no indication of how it was resolved.  Did someone actually find and fix the problem, or just change the error message as suggested in by Mark Thomas?

Thanks.
Comment 10 Konstantin Kolinko 2013-02-01 17:30:10 UTC
(In reply to comment #9)
1. There is no evidence that you have the same problem.

2. There is a number of ways to configure annotation scanning, or avoid it altogether in a specific application. The bonus point is that your application will start up faster. Bugzilla is not a support forum to go into details.

It is also possible to configure a different value for -Xss.

3. We are volunteers here. Feel free to propose a better patch than r1416502

Sometimes it is possible to change algorithm and unroll a recursion. I have not dug whether it is possible here.
Comment 11 John Bize 2013-02-01 20:29:06 UTC
(In reply to comment #10)

Thanks for your reply, and thanks for all the volunteer efforts on this project.  It is appreciated, and I didn't mean to imply otherwise.

I do believe I have the same problem.  When starting up 7.0.29 and deploying two WARs, one of my WARs fails to deploy with a very similar stack trace.  (I'm in deployWARs instead of deployDescriptors, but I think that must be due to a different version than the original post.  Other than that and the HashMap vs. HashSet, the stacks are pretty much the same.)  My stack ends with the following:

...
Caused by: java.lang.StackOverflowError at java.util.HashMap.get(HashMap.java:300) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2215) at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2227) at ...

There are 1024 recursions at line 2227 before the stack is exhausted.  I'm not able to set -Xss large enough to make any difference. The populateSCIsForCacheEntry always recurses 1024 times and throws the StackOverflowError Error.  That seems like a very deep recursion.  The fact that different deploy paths (<Host appBase=...) can change the behavior doesn't make sense either.

It isn't fixed for me, but I will try to find another resolution.  

Thanks again.
Comment 12 Mark Thomas 2013-02-01 20:34:03 UTC
The point here is that if you ship an application that does this:

<quote>
Due to dependency management problems I deployed a WAR file containing two JAR files containing different versions of a library, where version 1 contained a class A which inherited from B and another version 2 which contained a class B inheriting from A.
The classes were loaded in such an order that A and B cyclically inherited from each other which leads to a stack overflow in populateSCIsForCacheEntry because it does not detect cycles in the inheritance tree.
</quote>

Then the application is going to fall over as soon as it hits those classes. That is an application bug, not a Tomcat bug.

You might be lucky and not use those classes at all in your application. In which case you wouldn't see this until Tomcat was updated to scan classes for annotations as intended by the Servlet 3.0 EG. Hence it might look like a Tomcat problem.

If you have a valid set of JARs that can trigger a stack overflow then that would be a new bug and one that would get fairly prompt attention.
Comment 13 Aleksander 2013-02-12 09:07:17 UTC
I have the same problem in 7.0.35 version. My application is very big - over 200 jars, and it's hard to identify the jar which couse the problem. There is no information about classes in infinite loop call stack.

My call stack:


Caused by: java.lang.IllegalStateException: Unable to complete the scan for annotations for web application []. Possible root causes include a too low setting for -Xss and illegal cyclic inheritance dependencies
	at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2109)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1981)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1947)
	at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1932)
	at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1326)
	at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5179)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	... 11 more
Caused by: java.lang.StackOverflowError
	at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269)
	at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269)
	at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269)
Comment 14 Mark Thomas 2013-02-19 15:16:17 UTC
I'll see what I can do to add some class and/or JAR information to the error message.
Comment 15 Mark Thomas 2013-02-19 16:24:46 UTC
I have added the class hierarchy to the error message in trunk and 7.0.x. It will be included in 7.0.38 onwards.
Comment 16 Robert Pasztor 2013-03-21 10:22:40 UTC
I've tried to upgrade to 7.0.37 (latest release at this moment).
We still encounter the same issue, but this time the logged stacktrace is more detailed than the one I've added in the initial description:

10:49:06,990 ERROR [ContainerBase:903] ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/dossier]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:657)
        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1637)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: Unable to complete the scan for annotations for web application [/dossier]. Possible root causes include a
 too low setting for -Xss and illegal cyclic inheritance dependencies
        at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2109)
        at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1981)
        at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1947)
        at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1932)
        at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1326)
        at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5179)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 11 more
Caused by: java.lang.StackOverflowError
        at java.util.Random.nextInt(Random.java:239)
        at sun.misc.Hashing.randomHashSeed(Hashing.java:254)
        at java.util.HashMap.<init>(HashMap.java:255)
        at java.util.HashMap.<init>(HashMap.java:305)
        at java.util.HashSet.<init>(HashSet.java:103)
        at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2252)
        at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269)
        at org.apache.catalina.startup.ContextConfig.populateSCIsForCacheEntry(ContextConfig.java:2269)
...

I thought this might be helpful for further progress on this issue.
As mentioned in comment 10 and also on http://collab.sakaiproject.org/pipermail/sakai-dev/2012-November/019574.html ,

I was able to workaround this exception by skipping the jar scannining in catalina.properties:

# Additional JARs (over and above the default JARs listed above) to skip when
# scanning for Servlet 3.0 pluggability features. These features include web
# fragments, annotations, SCIs and classes that match @HandlesTypes. The list
# must be a comma separated list of JAR file names.
org.apache.catalina.startup.ContextConfig.jarsToSkip=*.jar

IMHO this is still a workaround and not the fix.
Comment 17 Joseph Walton 2013-03-26 09:03:51 UTC
Thanks for the new class hierarchy message. Running tomcat master with that improvement showed me exactly what the problem was - a loop caused by an incompatible version of Jaxen bundled with an old dom4j. Upgrading the dependencies in my webapp fixed it.
Comment 18 Tom 2013-11-18 21:22:23 UTC
FYI seems kind of weird but we have experienced this when for various reasons we needed to kill -9 the tomcat process. Doesn't necessarily make sense but was fairly consistent.