Created attachment 28703 [details] Changing the Hashmap by Collections.synchonizedMap() Not sure if it is java 7 related, but I had repeatetly that deadlock when I got the srver under some load. A simple replacement of the HashMap by a synchonized one did help (patch attached); but I dont know if that is a good solution as I dont get the idea of the AnnotationCache. http-apr-80-exec-348" daemon prio=10 tid=0x00007f4f64da9000 nid=0x6306 waiting for monitor entry [0x00007f4f46cd7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.catalina.core.DefaultInstanceManager.populateAnnotationsCache(DefaultInstanceManager.java:256) - waiting to lock <0x00000007063b3d90> (a java.util.WeakHashMap) at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:143) at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:137) at org.apache.jsp.templates.web.templates.s2.inc_005fcategory_jsp._jspx_meth_ox_005fifexists_005f1(inc_005fcategory_jsp.java:708) at org.apache.jsp.templates.web.templates.s2.inc_005fcategory_jsp._jspx_meth_ox_005frepeat_005f0(inc_005fcategory_jsp.java:118) at org.apache.jsp.templates.web.templates.s2.inc_005fcategory_jsp._jspService(inc_005fcategory_jsp.java:70) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
There is no evidence of a deadlock provided in this bug report. Please follow up on the users mailing list. Feel free to re-open this report once some evidence of a deadlock is available.
I was facing similar issue in my application where 272 threads are blocked by this monitor lock. Looks like using weak hash map is causing the issue. I have attached the screenshot of the dead lock situation. As Bob mentioned this issue is happening after certain user load. This issue is completely bringing down the application throughput.
Below is my current tomcat version, Server version: Apache Tomcat/7.0.42 Server built: Jul 2 2013 08:57:41 Server number: 7.0.42.0 OS Name: Linux OS Version: 2.6.32-431.23.3.el6.x86_64 Architecture: amd64 JVM Version: 1.7.0_71-mockbuild_2014_10_03_09_05-b00 JVM Vendor: Oracle Corporation
Created attachment 32293 [details] Monitor lock details
Created attachment 32294 [details] Method list
Again, there is no evidence of a deadlock in the new information provided. There is evidence of lock contention but no evidence as to what is causing that. Given that the sync blocks only surround get() and put(), I wonder if GC activity is the root cause. The stack trace for the thread that currently holds the lock on the annotationCache would be illuminating. This issue is going to remain as INVALID until some evidence of a Tomcat bug is produced.
Looks like this is similar to https://issues.apache.org/bugzilla/show_bug.cgi?id=53085
Concurrent hash map will solve this contention. The patch which Konstantin provided is for tomcat 8. Will it be possible to get it in tomcat 7?