We've seen crashes in production with our Tomcat 9 setup but we don't know what is causing it so it's hard to reproduce. Any help is appreciated. During startup: 22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.6.5]. 22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1a 20 Nov 2018] and the crash dump: # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000180060895, pid=7532, tid=0x0000000000000f88 # # JRE version: Java(TM) SE Runtime Environment (8.0_152-b16) (build 1.8.0_152-b16) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode windows-amd64 compressed oops) # Problematic frame: # C [tcnative-1.dll+0x60895] # # Core dump written. Default location: C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.16\hs_err_pid7532.mdmp # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- T H R E A D --------------- Current thread (0x000000004aff4800): JavaThread "https-openssl-apr-443-exec-80" daemon [_thread_in_native, id=3976, stack(0x000000004e8c0000,0x000000004e9c0000)] siginfo: ExceptionCode=0xc0000005, writing address 0x0000000000000278 Registers: RAX=0x00000001801f7120, RBX=0x000000004ce0cb70, RCX=0x0000000000000000, RDX=0x0000000000000040 RSP=0x000000004e9bf0f0, RBP=0x0000000180044610, RSI=0x0000000000000000, RDI=0x000000004ce0cb70 R8 =0x0000000000000002, R9 =0x000000004e9bf170, R10=0x0000000000000003, R11=0x0000000000000000 R12=0x0000000180044820, R13=0x000000018004a000, R14=0x0000000000000000, R15=0x000000004aff4800 RIP=0x0000000180060895, EFLAGS=0x0000000000010202 Top of Stack: (sp=0x000000004e9bf0f0) 0x000000004e9bf0f0: 0000000180044610 0000000000000000 0x000000004e9bf100: 000000004ce0cb70 0000000000000000 0x000000004e9bf110: 000000004e9bf180 000000004e9bf130 0x000000004e9bf120: 0000000100000000 0000000180044300 0x000000004e9bf130: 0000000000000000 00000001800860e5 0x000000004e9bf140: 0000000000000003 0000000180044610 0x000000004e9bf150: 000000004ce0cb70 0000000180010f60 0x000000004e9bf160: 0000000180044300 0000000180044781 0x000000004e9bf170: 0000000000000000 00000001801f7120 0x000000004e9bf180: 0000000000000000 0000000000000000 0x000000004e9bf190: 000000004ce0cb70 000000018003095b 0x000000004e9bf1a0: 000000004ce0cb70 0000000180044300 0x000000004e9bf1b0: 0000000180044610 0000000000000035 0x000000004e9bf1c0: 0000000180010f60 000000018003a3f5 0x000000004e9bf1d0: 000000004fc10890 0000000000000000 0x000000004e9bf1e0: 0000000000000035 0000000000000035 Instructions: (pc=0x0000000180060895) 0x0000000180060875: 5c 24 60 33 c0 48 83 c4 68 5f 5e c3 48 8b 8f a8 0x0000000180060885: 00 00 00 48 8b 84 24 88 00 00 00 48 89 6c 24 58 0x0000000180060895: 48 89 81 78 02 00 00 48 8b 84 24 98 00 00 00 48 0x00000001800608a5: 8b 8f a8 00 00 00 48 89 81 80 02 00 00 48 8b 8f Register to memory mapping: RAX=0x00000001801f7120 is an unknown value RBX=0x000000004ce0cb70 is an unknown value RCX=0x0000000000000000 is an unknown value RDX=0x0000000000000040 is an unknown value RSP=0x000000004e9bf0f0 is pointing into the stack for thread: 0x000000004aff4800 RBP=0x0000000180044610 is an unknown value RSI=0x0000000000000000 is an unknown value RDI=0x000000004ce0cb70 is an unknown value R8 =0x0000000000000002 is an unknown value R9 =0x000000004e9bf170 is pointing into the stack for thread: 0x000000004aff4800 R10=0x0000000000000003 is an unknown value R11=0x0000000000000000 is an unknown value R12=0x0000000180044820 is an unknown value R13=0x000000018004a000 is an unknown value R14=0x0000000000000000 is an unknown value R15=0x000000004aff4800 is a thread Stack: [0x000000004e8c0000,0x000000004e9c0000], sp=0x000000004e9bf0f0, free space=1020k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [tcnative-1.dll+0x60895] C [tcnative-1.dll+0x44781] C [tcnative-1.dll+0x3095b] C [tcnative-1.dll+0x31210] C [tcnative-1.dll+0x28dd9] C [tcnative-1.dll+0xff8e] C 0x0000000005694f03 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 21524 org.apache.tomcat.jni.SSLSocket.handshake(J)I (0 bytes) @ 0x0000000005694e81 [0x0000000005694e40+0x41] J 28413 C2 org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(Lorg/apache/tomcat/util/net/SocketWrapperBase;)Z (347 bytes) @ 0x0000000002bfe4a4 [0x0000000002bfe380+0x124] J 28085 C2 org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()V (199 bytes) @ 0x000000000694334c [0x0000000006943280+0xcc] J 47601 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x0000000002bb7014 [0x0000000002bb6960+0x6b4] J 46992 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x000000000986f424 [0x000000000986f3e0+0x44] j org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4 J 45601 C2 java.lang.Thread.run()V (17 bytes) @ 0x00000000024cb3f0 [0x00000000024cb3a0+0x50] v ~StubRoutines::call_stub Also some dynamic libs: Dynamic libraries: 0x000000013fad0000 - 0x000000013faef000 C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.16\bin\Tomcat9.0.16.exe 0x0000000076cf0000 - 0x0000000076e8f000 C:\windows\SYSTEM32\ntdll.dll 0x0000000076bd0000 - 0x0000000076cef000 C:\windows\system32\kernel32.dll 0x000007fefccb0000 - 0x000007fefcd1a000 C:\windows\system32\KERNELBASE.dll 0x000007fefd170000 - 0x000007fefd24b000 C:\windows\system32\ADVAPI32.dll 0x000007fefd7b0000 - 0x000007fefd84f000 C:\windows\system32\msvcrt.dll 0x000007fefefc0000 - 0x000007fefefdf000 C:\windows\SYSTEM32\sechost.dll 0x000007fefd500000 - 0x000007fefd62d000 C:\windows\system32\RPCRT4.dll 0x000007fefdc70000 - 0x000007fefe9fb000 C:\windows\system32\SHELL32.dll 0x000007fefd0d0000 - 0x000007fefd141000 C:\windows\system32\SHLWAPI.dll 0x000007fefd630000 - 0x000007fefd697000 C:\windows\system32\GDI32.dll 0x0000000076ad0000 - 0x0000000076bca000 C:\windows\system32\USER32.dll 0x000007fefcff0000 - 0x000007fefcffe000 C:\windows\system32\LPK.dll 0x000007fefd000000 - 0x000007fefd0cb000 C:\windows\system32\USP10.dll 0x000007fefedf0000 - 0x000007fefee1e000 C:\windows\system32\IMM32.DLL 0x000007fefd6a0000 - 0x000007fefd7a9000 C:\windows\system32\MSCTF.dll 0x00000000724c0000 - 0x0000000072d60000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\server\jvm.dll 0x000007fef9b40000 - 0x000007fef9b49000 C:\windows\system32\WSOCK32.dll 0x000007fefd430000 - 0x000007fefd47d000 C:\windows\system32\WS2_32.dll 0x000007fefdb30000 - 0x000007fefdb38000 C:\windows\system32\NSI.dll 0x000007fef9730000 - 0x000007fef976b000 C:\windows\system32\WINMM.dll 0x000007fefbd10000 - 0x000007fefbd1c000 C:\windows\system32\VERSION.dll 0x0000000076e90000 - 0x0000000076e97000 C:\windows\system32\PSAPI.DLL 0x00000000747a0000 - 0x0000000074872000 C:\windows\system32\MSVCR100.dll 0x0000000072260000 - 0x000000007226f000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\verify.dll 0x00000000721f0000 - 0x0000000072219000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\java.dll 0x0000000070e10000 - 0x0000000070e45000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\jdwp.dll 0x000000006ea10000 - 0x000000006ea18000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\npt.dll 0x0000000072240000 - 0x0000000072256000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\zip.dll 0x000007fefd930000 - 0x000007fefdb2d000 C:\windows\system32\ole32.dll 0x000007fefcbe0000 - 0x000007fefcbef000 C:\windows\system32\profapi.dll 0x000000006ea00000 - 0x000000006ea09000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\dt_socket.dll 0x000007fefc400000 - 0x000007fefc455000 C:\windows\system32\mswsock.dll 0x000007fefbde0000 - 0x000007fefbde7000 C:\windows\System32\wshtcpip.dll 0x0000000071650000 - 0x000000007165d000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\management.dll 0x0000000071550000 - 0x000000007156a000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\net.dll 0x000007fefc3f0000 - 0x000007fefc3f7000 C:\windows\System32\wship6.dll 0x0000000071530000 - 0x0000000071541000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\nio.dll 0x000007fefb520000 - 0x000007fefb535000 C:\windows\system32\NLAapi.dll 0x000007fefa410000 - 0x000007fefa425000 C:\windows\system32\napinsp.dll 0x000007fefc210000 - 0x000007fefc26b000 C:\windows\system32\DNSAPI.dll 0x000007fefa440000 - 0x000007fefa44b000 C:\windows\System32\winrnr.dll 0x000007fefaf70000 - 0x000007fefaf97000 C:\windows\system32\IPHLPAPI.DLL 0x000007fefaf10000 - 0x000007fefaf1b000 C:\windows\system32\WINNSI.DLL 0x000007fefa510000 - 0x000007fefa518000 C:\windows\system32\rasadhlp.dll 0x000007fefac80000 - 0x000007fefacd3000 C:\windows\System32\fwpuclnt.dll 0x000007fefc650000 - 0x000007fefc668000 C:\windows\system32\CRYPTSP.dll 0x000007fefc080000 - 0x000007fefc0c7000 C:\windows\system32\rsaenh.dll 0x000007fefcad0000 - 0x000007fefcadf000 C:\windows\system32\CRYPTBASE.dll 0x000007fefabf0000 - 0x000007fefac01000 C:\windows\system32\dhcpcsvc6.DLL 0x000007fefaab0000 - 0x000007fefaac8000 C:\windows\system32\dhcpcsvc.DLL 0x0000000180000000 - 0x0000000180281000 C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.16\bin\tcnative-1.dll 0x000007fefce00000 - 0x000007fefcf6d000 C:\windows\system32\CRYPT32.dll 0x000007fefcc80000 - 0x000007fefcc8f000 C:\windows\system32\MSASN1.dll 0x000007fefc590000 - 0x000007fefc5b2000 C:\windows\system32\bcrypt.dll 0x000007fefc030000 - 0x000007fefc07c000 C:\windows\system32\bcryptprimitives.dll 0x00000000709b0000 - 0x00000000709d4000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\sunec.dll 0x00000000709a0000 - 0x00000000709ab000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\sunmscapi.dll 0x000000006f380000 - 0x000000006f518000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\awt.dll 0x000007fefd850000 - 0x000007fefd92a000 C:\windows\system32\OLEAUT32.dll 0x000007fefca70000 - 0x000007fefcac7000 C:\windows\system32\apphelp.dll 0x0000000070d90000 - 0x0000000070dc1000 C:\Program Files\Java\jdk1.8.0_152\jre\bin\jpeg.dll 0x000007feface0000 - 0x000007fefacf8000 C:\windows\system32\DWMAPI.DLL 0x000007fef98e0000 - 0x000007fef9a05000 C:\windows\system32\DBGHELP.DLL
There are some changes in 9.0.21 onwards that might address this. If you are able to upgrade and retest that would be helpful.
Does this still occur with the most recent versions of Tomcat, libtcnative and libapr?
No response for over a year. I am going to assume that changes in 9.0.21 addressed this. If you still see this issue or something similar please: - update to the latest 9.0.x release - update to the latest Tomcat Native release - retest - if you still see the issue, feel free to re-open this bug What we really need are the steps to reproduce it. Anything that narrows down the trigger is helpful but a set of steps to reproduce is ideal.