Bug 52141 - [STUCK] ExecuteThread...Blocked trying to get lock: org/apache/log4j/Logger@0xc501e0a8[fat lock]
Summary: [STUCK] ExecuteThread...Blocked trying to get lock: org/apache/log4j/Logger@0...
Status: NEW
Alias: None
Product: Log4j - Now in Jira
Classification: Unclassified
Component: Appender (show other bugs)
Version: 1.2
Hardware: Sun Linux
: P2 blocker
Target Milestone: ---
Assignee: log4j-dev
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-05 18:20 UTC by sreehari p
Modified: 2011-11-05 18:24 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description sreehari p 2011-11-05 18:20:11 UTC
Hi,

My application is getting stuck with threads and going very frequently to overloaded state. When this happens, we have to restart the server to void this issue. Since we dont know when this problem occurs, our support team has to keep monitor the server all the time. 

The environment details are:

Web Logic - 10.3
Java - 6.0
Log4j - log4j-1.2.15

Just to let you know that we use log4j to write the application logs into DB. I am thinking that it might be causing an issue but not sure. Here am sending the thread dump from one of our application server. If you could help what is the an issue, it would be great help.

Thanks for your help in advance

Thanks
Sree Hari
Comment 1 sreehari p 2011-11-05 18:24:41 UTC
Also, we have seen same overloaded state issue on nov 3rd. Here is the thread dump taken when issue occured.

            "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native
          
            	jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
          
            	jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
          
            	java.net.SocketInputStream.socketRead0(SocketInputStream.java)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
          
            	com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
          
            	com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
          
            	com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
          
            	com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
          
            	java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          
            	java.io.BufferedInputStream.read(BufferedInputStream.java:237)
          
            	org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
          
            	org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
          
            	org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
          
            	org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
          
            	org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
          
            	org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
          
            	org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
          
            	org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
          
            	org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
          
            	org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
          
            	com.bestbuy.bbym.beast.cap.core.http.HttpSender.sendRequestAndFetchResponse(HttpSender.java:67)
          
            	com.bestbuy.bbym.beast.cap.core.http.HttpAdapter.processRequest(HttpAdapter.java:164)
          
            	com.bestbuy.bbym.beast.cap.core.ProxyHelper.getResponse(ProxyHelper.java:324)
          
            	com.bestbuy.bbym.beast.cap.core.multirequest.PostRequest.execute(PostRequest.java:237)
          
            	com.bestbuy.bbym.beast.cap.util.MultiRequestUtil.executeNextCommand(MultiRequestUtil.java:185)
          
            	com.bestbuy.bbym.beast.cap.core.multirequest.MultiRequestProxy.invoke(MultiRequestProxy.java:87)
          
            	com.bestbuy.bbym.beast.cap.core.RequestProcessor.process(RequestProcessor.java:68)
          
            	com.bestbuy.bbym.beast.cap.core.controller.Controller.handleRequest(Controller.java:737)
          
            	com.bestbuy.bbym.beast.cap.core.controller.SyncServiceHandlingController.handleSyncRequest(SyncServiceHandlingController.java:28)
          
            	com.bestbuy.bbym.beast.cap.core.router.RequestRouter.routeRequest(RequestRouter.java:71)
          
            	com.bestbuy.bbym.beast.cap.core.controller.CAPWsController.processRequest(CAPWsController.java:68)
          
            	sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
          
            	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          
            	java.lang.reflect.Method.invoke(Method.java:597)
          
            	org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
          
            	org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
          
            	org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
          
            	org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
          
            	org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
          
            	org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
          
            	org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
          
            	org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
          
            	org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
          
            	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
          
            	org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
          
            	javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
          
            	weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
          
            	weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
          
            	weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
          
            	weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
          
            	weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3495)
          
            	weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
          
            	weblogic.security.service.SecurityManager.runAs(Unknown Source)
          
            	weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
          
            	weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
          
            	weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
          
            	weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
          
            "[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@16ae0c72 WAITING
          
            	java.lang.Object.wait(Object.java:485)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:157)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
          
            "[STANDBY] ExecuteThread: '91' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1653e8a3 WAITING
          
            	java.lang.Object.wait(Object.java:485)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:157)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
          
            "Thread-78" RUNNABLE native
          
            	jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
          
            	jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
          
            	java.net.SocketInputStream.socketRead0(SocketInputStream.java)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
          
            	com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
          
            	com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
          
            	com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
          
            	com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
          
            	java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          
            	java.io.BufferedInputStream.read(BufferedInputStream.java:237)
          
            	org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
          
            	org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
          
            	org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
          
            	org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
          
            	org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
          
            	org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
          
            	org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
          
            	org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
          
            	org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
          
            	org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
          
            	com.bestbuy.bbym.beast.cap.core.http.HttpSender.sendRequestAndFetchResponse(HttpSender.java:67)
          
            	com.bestbuy.bbym.beast.cap.core.http.HttpAdapter.processRequest(HttpAdapter.java:164)
          
            	com.bestbuy.bbym.beast.cap.core.ProxyHelper.getResponse(ProxyHelper.java:324)
          
            	com.bestbuy.bbym.beast.cap.core.ProxyHelper$1.call(ProxyHelper.java:151)
          
            	com.bestbuy.bbym.beast.cap.core.ProxyHelper$1.call(ProxyHelper.java:1)
          
            	java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          
            	java.util.concurrent.FutureTask.run(FutureTask.java:138)
          
            	java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          
            	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          
            	java.lang.Thread.run(Thread.java:619)
          
            "LDAPConnThread-19 ldaps://directory.bestbuy.com:636" RUNNABLE native
          
            	jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
          
            	jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
          
            	java.net.SocketInputStream.socketRead0(SocketInputStream.java)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:159)
          
            	java.io.InputStream.read(InputStream.java:85)
          
            	com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
          
            	com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
          
            	com.certicom.tls.record.ReadHandler.read(Unknown Source)
          
            	com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
          
            	java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          
            	java.io.BufferedInputStream.read(BufferedInputStream.java:237)
          
            	netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
          
            	netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
          
            	java.lang.Thread.run(Thread.java:619)
          
            "LDAPConnThread-20 ldaps://directory.bestbuy.com:636" RUNNABLE native
          
            	jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
          
            	jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
          
            	java.net.SocketInputStream.socketRead0(SocketInputStream.java)
          
            	java.net.SocketInputStream.read(SocketInputStream.java:129)
          
            	weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:159)
          
            	java.io.InputStream.read(InputStream.java:85)
          
            	com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
          
            	com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
          
            	com.certicom.tls.record.ReadHandler.read(Unknown Source)
          
            	com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
          
            	java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          
            	java.io.BufferedInputStream.read(BufferedInputStream.java:237)
          
            	netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
          
            	netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
          
            	java.lang.Thread.run(Thread.java:619)
          
            "[STANDBY] ExecuteThread: '66' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1121a634 WAITING
          
            	java.lang.Object.wait(Object.java:485)
          
            	weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:157)
          
            	weblogic.work.ExecuteThread.run(ExecuteThread.java:178)