Bug 53171 - Deadlock under Java 7 / DefaultInstanceManager
Summary: Deadlock under Java 7 / DefaultInstanceManager
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 7.0.27
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2012-05-01 10:58 UTC by Bob
Modified: 2014-12-17 07:46 UTC (History)
0 users

Changing the Hashmap by Collections.synchonizedMap() (4.26 KB, application/octet-stream)
2012-05-01 10:58 UTC, Bob
Monitor lock details (90.55 KB, image/png)
2014-12-16 09:09 UTC, Manoj
Method list (11.94 KB, image/png)
2014-12-16 09:09 UTC, Manoj

Note You need to log in before you can comment on or make changes to this bug.
Description Bob 2012-05-01 10:58:32 UTC
Created attachment 28703 [details]
Changing the Hashmap by Collections.synchonizedMap()

Not sure if it is java 7 related, but I had repeatetly that deadlock when I got the srver under some load. A simple replacement of the HashMap by a synchonized one did help (patch attached); but I dont know if that is a good solution as I dont get the idea of the AnnotationCache.

http-apr-80-exec-348" daemon prio=10 tid=0x00007f4f64da9000 nid=0x6306 waiting for monitor entry [0x00007f4f46cd7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.catalina.core.DefaultInstanceManager.populateAnnotationsCache(DefaultInstanceManager.java:256)
	- waiting to lock <0x00000007063b3d90> (a java.util.WeakHashMap)
	at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:143)
	at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:137)
	at org.apache.jsp.templates.web.templates.s2.inc_005fcategory_jsp._jspx_meth_ox_005fifexists_005f1(inc_005fcategory_jsp.java:708)
	at org.apache.jsp.templates.web.templates.s2.inc_005fcategory_jsp._jspx_meth_ox_005frepeat_005f0(inc_005fcategory_jsp.java:118)
	at org.apache.jsp.templates.web.templates.s2.inc_005fcategory_jsp._jspService(inc_005fcategory_jsp.java:70)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
Comment 1 Mark Thomas 2012-05-29 13:29:55 UTC
There is no evidence of a deadlock provided in this bug report. Please follow up on the users mailing list. Feel free to re-open this report once some evidence of a deadlock is available.
Comment 2 Manoj 2014-12-16 09:04:23 UTC
I was facing similar issue in my application where 272 threads are blocked by this monitor lock. Looks like using weak hash map is causing the issue. I have attached the screenshot of the dead lock situation. As Bob mentioned this issue is happening after certain user load.

This issue is completely bringing down the application throughput.
Comment 3 Manoj 2014-12-16 09:05:22 UTC
Below is my current tomcat version,

Server version: Apache Tomcat/7.0.42
Server built:   Jul 2 2013 08:57:41
Server number:
OS Name:        Linux
OS Version:     2.6.32-431.23.3.el6.x86_64
Architecture:   amd64
JVM Version:    1.7.0_71-mockbuild_2014_10_03_09_05-b00
JVM Vendor:     Oracle Corporation
Comment 4 Manoj 2014-12-16 09:09:11 UTC
Created attachment 32293 [details]
Monitor lock details
Comment 5 Manoj 2014-12-16 09:09:55 UTC
Created attachment 32294 [details]
Method list
Comment 6 Mark Thomas 2014-12-16 23:07:17 UTC
Again, there is no evidence of a deadlock in the new information provided.

There is evidence of lock contention but no evidence as to what is causing that. Given that the sync blocks only surround get() and put(), I wonder if GC activity is the root cause.

The stack trace for the thread that currently holds the lock on the annotationCache would be illuminating.

This issue is going to remain as INVALID until some evidence of a Tomcat bug is produced.
Comment 7 Manoj 2014-12-17 07:02:19 UTC
Looks like this is similar to https://issues.apache.org/bugzilla/show_bug.cgi?id=53085
Comment 8 Manoj 2014-12-17 07:46:56 UTC
Concurrent hash map will solve this contention. The patch which Konstantin provided is for tomcat 8. Will it be possible to get it in tomcat 7?