Created attachment 30877 [details] Crash report attached Apache Tomcat/7.0.29 APR based Apache Tomcat Native library 1.1.24 using APR version 1.3.9. Tomcat with the APR connector, using HTTPS scheme. The system is under load test with heavy traffic flowing between client and server. It also occurred once when the system was NOT under load. No known test case to reproduce the issue. It happens randomly. More information in case it helps: There are around 100 clients connecting to the server from the same machine. There are 3 such machines on which clients run. https://issues.apache.org/bugzilla/show_bug.cgi?id=51813 looks similar but probably not same issue. # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f0df4825e39, pid=1208, tid=139688203269888 # # JRE version: 7.0_03 # Java VM: OpenJDK 64-Bit Server VM (22.0-b10 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libtcnative-1.so+0x12e39] Java_org_apache_tomcat_jni_Socket_sendbb+0x59 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please include # instructions on how to reproduce the bug and visit: # http://icedtea.classpath.org/bugzilla # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- T H R E A D --------------- Current thread (0x00007f0c6c151800): JavaThread "http-apr-9443-exec-277" daemon [_thread_in_native, id=6009, stack(0x00007f0bb1b43000,0x00007f0bb1bc4000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000040 . . . Stack: [0x00007f0bb1b43000,0x00007f0bb1bc4000], sp=0x00007f0bb1bc20d0, free space=508k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libtcnative-1.so+0x12e39] Java_org_apache_tomcat_jni_Socket_sendbb+0x59 [error occurred during error reporting (printing native stack), id 0xb] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J org.apache.tomcat.jni.Socket.sendbb(JII)I J org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V J org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V J net.zschech.gwt.comet.server.deflate.DeflaterOutputStream.flush()V J sun.nio.cs.StreamEncoder.flush()V J net.zschech.gwt.comet.server.impl.CometServletResponseImpl.tryTerminate()V J net.zschech.gwt.comet.server.impl.CometServletResponseImpl.initiate()V J javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V J javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V J org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; J org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; J org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()V J java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub
Both Tomcat and tcnative are out of date. Can you re-try your testing with current versions (current are Tomcat 7.0.42, tcnative 1.1.27). Also, APR 1.3.9 is fairly old. What type of Linux system is this? Also, please run 'ulimit -c unlimited' in the shell before launching Tomcat so you can get a core file -- hopefully with a more complete backtrace.
Also, why do you believe this is not the issue covered by bug #51813 ?
The ulimit is currently set to 10000 for this tomcat process. The file handler count has hardly gone beyond 3000. I will test with 'ulimit -c unlimited' to capture core dump in case this occurs again. The difference between this issue and bug #51813 is the data between server and client is purely text based wheres the bug you mentioned talks about some issue related to image IO. In my case, there is no image IO. Regarding using latest version and reproducing: There is no known test case to reproduce the issue. It happened only twice in around 100 load test runs. Sorry but I won't be able to test using the versions mentioned by you.
What could be side effects of running tomcat with 'ulimit -c unlimited' during the test?
Created attachment 31011 [details] Another crash with same error There was another crash and I have attached the crash report file hs_err_pid12092.log. There is no change to any of the versions/component/system/environment etc.
(In reply to Nitinb from comment #4) > What could be side effects of running tomcat with 'ulimit -c unlimited' > during the test? You might get a very large core file (roughly the size of your JVM process).
(In reply to Nitinb from comment #5) > Created attachment 31011 [details] > Another crash with same error > > There was another crash and I have attached the crash report file > hs_err_pid12092.log. There is no change to any of the > versions/component/system/environment etc. Exactly what version of tcnative were you running at the time? I've already mentioned that both your Tomcat and tcnative are out of date, and I believe this issue to have been resolved as bug #51813. The reason why that other bug is related to this one is because the output stream is stale and the sendbb function doesn't check for that case. There have been many patches to both Tomcat and tcnative that should handle the issue you describe here. Please upgrade to new versions of both components and re-test.
Without further information, this bug will be marked as WORKSFORME soon.
After upgrading tomcat to 7.47 and APR to apr-1.4.6 and so far didn't encounter the issue. Kindly mark it as resolved. I will reopen in case the same issue is encountered. Thanks.