Bug 45516

Summary:	All Tomcat http threads stuck in java.net.SocketOutputStream.socketWrite0(Native Method)
Product:	Tomcat 6	Reporter:	Ville Hartikainen <ville.hartikainen>
Component:	Catalina	Assignee:	Tomcat Developers Mailing List <dev>
Status:	RESOLVED INVALID
Severity:	normal
Priority:	P2
Version:	6.0.14
Target Milestone:	default
Hardware:	Other
OS:	Linux

Description Ville Hartikainen 2008-07-31 05:28:14 UTC

Between random intervals our website running with Liferay + Tomcat 6.0.14 gets stuck. 

CPU utilization is close to nothing. After getting a thread dump from the running java process, a following stacktraces are seen in all of the http-threads:

Is anybody able to tell, whether this is a bug in Tomcat, JVM, in our configuration or have I just diagnosed the problem incorrectly? 

Something to workaround this problem?
- - -
"http-9095-28" daemon prio=1 tid=0x0000000044743190 nid=0x3bc runnable [0x000000005124a000..0x000000005124be10]
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
	at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:761)
	at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:118)
	at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
	at org.apache.coyote.Response.doWrite(Response.java:560)
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)
	at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381)
	at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)
	at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
	- locked <0x00002aaaf85f8ac0> (a java.io.BufferedOutputStream)
	at com.liferay.util.servlet.ServletResponseUtil.write(ServletResponseUtil.java:194)
	at com.liferay.filters.strip.StripFilter.doFilter(StripFilter.java:249)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.filters.secure.SecureFilter.doFilter(SecureFilter.java:144)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.filters.compression.CompressionFilter.doFilter(CompressionFilter.java:122)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.filters.doubleclick.DoubleClickFilter.doFilter(DoubleClickFilter.java:135)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.servlet.filters.layoutcache.LayoutCacheFilter.doFilter(LayoutCacheFilter.java:167)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.doFilter(AutoLoginFilter.java:106)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:147)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292)
	at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:197)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
	at java.lang.Thread.run(Thread.java:595)

Comment 1 Ville Hartikainen 2008-08-01 00:52:56 UTC

Version information of used JDK and Tomcat, as reported by "catalina.sh version"
------------------
Using JRE_HOME:       /usr/java/jdk1.5.0_13/
Server version: Apache Tomcat/6.0.14
Server built:   Jul 20 2007 04:17:30
Server number:  6.0.14.0
OS Name:        Linux
OS Version:     2.6.18-53.1.4.el5
Architecture:   amd64
JVM Version:    1.5.0_13-b05
JVM Vendor:     Sun Microsystems Inc.

Comment 2 Mark Thomas 2008-08-01 07:47:17 UTC

This occasionally gets reported on the users list. The most recent occurrence appears to be this thread:
http://markmail.org/message/hghfhwoxdoa6fg5s

My best guess at the moment as to the root cause is some form of network oddity.

The users list is the best place to get help with this issue as bugzilla is really suited to being used as a support forum.

If you investigations identify a bug in Tomcat, feel free to re-open this issue and provide the appropriate information.