Bug 55588

Summary: Tomcat randomly crashes with [libtcnative-1.so+0x12e39] Java_org_apache_tomcat_jni_Socket_sendbb+0x59
Product: Tomcat Native Reporter: Nitinb <nitinvbhavsar>
Component: LibraryAssignee: Tomcat Developers Mailing List <dev>
Severity: major CC: nitinvbhavsar, st.mailinglists
Priority: P2    
Version: 1.1.24   
Target Milestone: ---   
Hardware: HP   
OS: Linux   
Attachments: Crash report attached
Another crash with same error

Description Nitinb 2013-09-24 06:06:46 UTC
Created attachment 30877 [details]
Crash report attached

Apache Tomcat/7.0.29
APR based Apache Tomcat Native library 1.1.24 using APR version 1.3.9.

Tomcat with the APR connector, using HTTPS scheme. The system is under load test with heavy traffic flowing between client and server. It also occurred once when the system was NOT under load. No known test case to reproduce the issue. It happens randomly. 
More information in case it helps: There are around 100 clients connecting to the server from the same machine. There are 3 such machines on which clients run.

https://issues.apache.org/bugzilla/show_bug.cgi?id=51813 looks similar but probably not same issue.

# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x00007f0df4825e39, pid=1208, tid=139688203269888
# JRE version: 7.0_03
# Java VM: OpenJDK 64-Bit Server VM (22.0-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libtcnative-1.so+0x12e39]  Java_org_apache_tomcat_jni_Socket_sendbb+0x59
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

---------------  T H R E A D  ---------------

Current thread (0x00007f0c6c151800):  JavaThread "http-apr-9443-exec-277" daemon [_thread_in_native, id=6009, stack(0x00007f0bb1b43000,0x00007f0bb1bc4000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000040
Stack: [0x00007f0bb1b43000,0x00007f0bb1bc4000],  sp=0x00007f0bb1bc20d0,  free space=508k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtcnative-1.so+0x12e39]  Java_org_apache_tomcat_jni_Socket_sendbb+0x59

[error occurred during error reporting (printing native stack), id 0xb]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  org.apache.tomcat.jni.Socket.sendbb(JII)I
J  org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V
J  org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
J  net.zschech.gwt.comet.server.deflate.DeflaterOutputStream.flush()V
J  sun.nio.cs.StreamEncoder.flush()V
J  net.zschech.gwt.comet.server.impl.CometServletResponseImpl.tryTerminate()V
J  net.zschech.gwt.comet.server.impl.CometServletResponseImpl.initiate()V
J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J  org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
J  org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V
J  org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
J  org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
J  org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()V
J  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
Comment 1 Christopher Schultz 2013-09-27 14:48:32 UTC
Both Tomcat and tcnative are out of date. Can you re-try your testing with current versions (current are Tomcat 7.0.42, tcnative 1.1.27). Also, APR 1.3.9 is fairly old. What type of Linux system is this?

Also, please run 'ulimit -c unlimited' in the shell before launching Tomcat so you can get a core file -- hopefully with a more complete backtrace.
Comment 2 Christopher Schultz 2013-09-27 14:50:54 UTC
Also, why do you believe this is not the issue covered by bug #51813 ?
Comment 3 Nitinb 2013-10-21 12:00:13 UTC
The ulimit is currently set to 10000 for this tomcat process. The file handler count has hardly gone beyond 3000. I will test with 'ulimit -c unlimited' to capture core dump in case this occurs again.

The difference between this issue and bug #51813 is the data between server and client is purely text based wheres the bug you mentioned talks about some issue related to image IO. In my case, there is no image IO.

Regarding using latest version and reproducing:
There is no known test case to reproduce the issue. It happened only twice in around 100 load test runs. Sorry but I won't be able to test using the versions mentioned by you.
Comment 4 Nitinb 2013-11-05 13:04:37 UTC
What could be side effects of running tomcat with 'ulimit -c unlimited' during the test?
Comment 5 Nitinb 2013-11-05 14:12:41 UTC
Created attachment 31011 [details]
Another crash with same error

There was another crash and I have attached the crash report file hs_err_pid12092.log. There is no change to any of the versions/component/system/environment etc.
Comment 6 Christopher Schultz 2013-11-05 14:41:06 UTC
(In reply to Nitinb from comment #4)
> What could be side effects of running tomcat with 'ulimit -c unlimited'
> during the test?

You might get a very large core file (roughly the size of your JVM process).
Comment 7 Christopher Schultz 2013-11-05 16:55:58 UTC
(In reply to Nitinb from comment #5)
> Created attachment 31011 [details]
> Another crash with same error
> There was another crash and I have attached the crash report file
> hs_err_pid12092.log. There is no change to any of the
> versions/component/system/environment etc.

Exactly what version of tcnative were you running at the time?

I've already mentioned that both your Tomcat and tcnative are out of date, and I believe this issue to have been resolved as bug #51813.

The reason why that other bug is related to this one is because the output stream is stale and the sendbb function doesn't check for that case. There have been many patches to both Tomcat and tcnative that should handle the issue you describe here. Please upgrade to new versions of both components and re-test.
Comment 8 Christopher Schultz 2013-12-20 03:28:23 UTC
Without further information, this bug will be marked as WORKSFORME soon.
Comment 9 Nitinb 2014-01-02 11:50:28 UTC
After upgrading tomcat to 7.47 and APR to apr-1.4.6 and so far didn't encounter the issue.
Kindly mark it as resolved. I will reopen in case the same issue is encountered.