Bug 63199 - sslsocket handshake JVM crash
Summary: sslsocket handshake JVM crash
Status: NEW
Alias: None
Product: Tomcat Native
Classification: Unclassified
Component: Library (show other bugs)
Version: 1.2.21
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-02-22 12:58 UTC by alexander.magnusson
Modified: 2019-02-22 13:11 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description alexander.magnusson 2019-02-22 12:58:29 UTC
We've seen crashes in production with our Tomcat 9 setup but we don't know what is causing it so it's hard to reproduce.
Any help is appreciated.

During startup:

22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.6.5].
22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
22-Feb-2019 12:44:04.253 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1a  20 Nov 2018]

and the crash dump:


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000180060895, pid=7532, tid=0x0000000000000f88
#
# JRE version: Java(TM) SE Runtime Environment (8.0_152-b16) (build 1.8.0_152-b16)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [tcnative-1.dll+0x60895]
#
# Core dump written. Default location: C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.16\hs_err_pid7532.mdmp
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x000000004aff4800):  JavaThread "https-openssl-apr-443-exec-80" daemon [_thread_in_native, id=3976, stack(0x000000004e8c0000,0x000000004e9c0000)]

siginfo: ExceptionCode=0xc0000005, writing address 0x0000000000000278

Registers:
RAX=0x00000001801f7120, RBX=0x000000004ce0cb70, RCX=0x0000000000000000, RDX=0x0000000000000040
RSP=0x000000004e9bf0f0, RBP=0x0000000180044610, RSI=0x0000000000000000, RDI=0x000000004ce0cb70
R8 =0x0000000000000002, R9 =0x000000004e9bf170, R10=0x0000000000000003, R11=0x0000000000000000
R12=0x0000000180044820, R13=0x000000018004a000, R14=0x0000000000000000, R15=0x000000004aff4800
RIP=0x0000000180060895, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x000000004e9bf0f0)
0x000000004e9bf0f0:   0000000180044610 0000000000000000
0x000000004e9bf100:   000000004ce0cb70 0000000000000000
0x000000004e9bf110:   000000004e9bf180 000000004e9bf130
0x000000004e9bf120:   0000000100000000 0000000180044300
0x000000004e9bf130:   0000000000000000 00000001800860e5
0x000000004e9bf140:   0000000000000003 0000000180044610
0x000000004e9bf150:   000000004ce0cb70 0000000180010f60
0x000000004e9bf160:   0000000180044300 0000000180044781
0x000000004e9bf170:   0000000000000000 00000001801f7120
0x000000004e9bf180:   0000000000000000 0000000000000000
0x000000004e9bf190:   000000004ce0cb70 000000018003095b
0x000000004e9bf1a0:   000000004ce0cb70 0000000180044300
0x000000004e9bf1b0:   0000000180044610 0000000000000035
0x000000004e9bf1c0:   0000000180010f60 000000018003a3f5
0x000000004e9bf1d0:   000000004fc10890 0000000000000000
0x000000004e9bf1e0:   0000000000000035 0000000000000035 

Instructions: (pc=0x0000000180060895)
0x0000000180060875:   5c 24 60 33 c0 48 83 c4 68 5f 5e c3 48 8b 8f a8
0x0000000180060885:   00 00 00 48 8b 84 24 88 00 00 00 48 89 6c 24 58
0x0000000180060895:   48 89 81 78 02 00 00 48 8b 84 24 98 00 00 00 48
0x00000001800608a5:   8b 8f a8 00 00 00 48 89 81 80 02 00 00 48 8b 8f 


Register to memory mapping:

RAX=0x00000001801f7120 is an unknown value
RBX=0x000000004ce0cb70 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x0000000000000040 is an unknown value
RSP=0x000000004e9bf0f0 is pointing into the stack for thread: 0x000000004aff4800
RBP=0x0000000180044610 is an unknown value
RSI=0x0000000000000000 is an unknown value
RDI=0x000000004ce0cb70 is an unknown value
R8 =0x0000000000000002 is an unknown value
R9 =0x000000004e9bf170 is pointing into the stack for thread: 0x000000004aff4800
R10=0x0000000000000003 is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x0000000180044820 is an unknown value
R13=0x000000018004a000 is an unknown value
R14=0x0000000000000000 is an unknown value
R15=0x000000004aff4800 is a thread


Stack: [0x000000004e8c0000,0x000000004e9c0000],  sp=0x000000004e9bf0f0,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [tcnative-1.dll+0x60895]
C  [tcnative-1.dll+0x44781]
C  [tcnative-1.dll+0x3095b]
C  [tcnative-1.dll+0x31210]
C  [tcnative-1.dll+0x28dd9]
C  [tcnative-1.dll+0xff8e]
C  0x0000000005694f03

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 21524  org.apache.tomcat.jni.SSLSocket.handshake(J)I (0 bytes) @ 0x0000000005694e81 [0x0000000005694e40+0x41]
J 28413 C2 org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(Lorg/apache/tomcat/util/net/SocketWrapperBase;)Z (347 bytes) @ 0x0000000002bfe4a4 [0x0000000002bfe380+0x124]
J 28085 C2 org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()V (199 bytes) @ 0x000000000694334c [0x0000000006943280+0xcc]
J 47601 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x0000000002bb7014 [0x0000000002bb6960+0x6b4]
J 46992 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x000000000986f424 [0x000000000986f3e0+0x44]
j  org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4
J 45601 C2 java.lang.Thread.run()V (17 bytes) @ 0x00000000024cb3f0 [0x00000000024cb3a0+0x50]
v  ~StubRoutines::call_stub



Also some dynamic libs:

Dynamic libraries:
0x000000013fad0000 - 0x000000013faef000 	C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.16\bin\Tomcat9.0.16.exe
0x0000000076cf0000 - 0x0000000076e8f000 	C:\windows\SYSTEM32\ntdll.dll
0x0000000076bd0000 - 0x0000000076cef000 	C:\windows\system32\kernel32.dll
0x000007fefccb0000 - 0x000007fefcd1a000 	C:\windows\system32\KERNELBASE.dll
0x000007fefd170000 - 0x000007fefd24b000 	C:\windows\system32\ADVAPI32.dll
0x000007fefd7b0000 - 0x000007fefd84f000 	C:\windows\system32\msvcrt.dll
0x000007fefefc0000 - 0x000007fefefdf000 	C:\windows\SYSTEM32\sechost.dll
0x000007fefd500000 - 0x000007fefd62d000 	C:\windows\system32\RPCRT4.dll
0x000007fefdc70000 - 0x000007fefe9fb000 	C:\windows\system32\SHELL32.dll
0x000007fefd0d0000 - 0x000007fefd141000 	C:\windows\system32\SHLWAPI.dll
0x000007fefd630000 - 0x000007fefd697000 	C:\windows\system32\GDI32.dll
0x0000000076ad0000 - 0x0000000076bca000 	C:\windows\system32\USER32.dll
0x000007fefcff0000 - 0x000007fefcffe000 	C:\windows\system32\LPK.dll
0x000007fefd000000 - 0x000007fefd0cb000 	C:\windows\system32\USP10.dll
0x000007fefedf0000 - 0x000007fefee1e000 	C:\windows\system32\IMM32.DLL
0x000007fefd6a0000 - 0x000007fefd7a9000 	C:\windows\system32\MSCTF.dll
0x00000000724c0000 - 0x0000000072d60000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\server\jvm.dll
0x000007fef9b40000 - 0x000007fef9b49000 	C:\windows\system32\WSOCK32.dll
0x000007fefd430000 - 0x000007fefd47d000 	C:\windows\system32\WS2_32.dll
0x000007fefdb30000 - 0x000007fefdb38000 	C:\windows\system32\NSI.dll
0x000007fef9730000 - 0x000007fef976b000 	C:\windows\system32\WINMM.dll
0x000007fefbd10000 - 0x000007fefbd1c000 	C:\windows\system32\VERSION.dll
0x0000000076e90000 - 0x0000000076e97000 	C:\windows\system32\PSAPI.DLL
0x00000000747a0000 - 0x0000000074872000 	C:\windows\system32\MSVCR100.dll
0x0000000072260000 - 0x000000007226f000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\verify.dll
0x00000000721f0000 - 0x0000000072219000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\java.dll
0x0000000070e10000 - 0x0000000070e45000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\jdwp.dll
0x000000006ea10000 - 0x000000006ea18000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\npt.dll
0x0000000072240000 - 0x0000000072256000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\zip.dll
0x000007fefd930000 - 0x000007fefdb2d000 	C:\windows\system32\ole32.dll
0x000007fefcbe0000 - 0x000007fefcbef000 	C:\windows\system32\profapi.dll
0x000000006ea00000 - 0x000000006ea09000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\dt_socket.dll
0x000007fefc400000 - 0x000007fefc455000 	C:\windows\system32\mswsock.dll
0x000007fefbde0000 - 0x000007fefbde7000 	C:\windows\System32\wshtcpip.dll
0x0000000071650000 - 0x000000007165d000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\management.dll
0x0000000071550000 - 0x000000007156a000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\net.dll
0x000007fefc3f0000 - 0x000007fefc3f7000 	C:\windows\System32\wship6.dll
0x0000000071530000 - 0x0000000071541000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\nio.dll
0x000007fefb520000 - 0x000007fefb535000 	C:\windows\system32\NLAapi.dll
0x000007fefa410000 - 0x000007fefa425000 	C:\windows\system32\napinsp.dll
0x000007fefc210000 - 0x000007fefc26b000 	C:\windows\system32\DNSAPI.dll
0x000007fefa440000 - 0x000007fefa44b000 	C:\windows\System32\winrnr.dll
0x000007fefaf70000 - 0x000007fefaf97000 	C:\windows\system32\IPHLPAPI.DLL
0x000007fefaf10000 - 0x000007fefaf1b000 	C:\windows\system32\WINNSI.DLL
0x000007fefa510000 - 0x000007fefa518000 	C:\windows\system32\rasadhlp.dll
0x000007fefac80000 - 0x000007fefacd3000 	C:\windows\System32\fwpuclnt.dll
0x000007fefc650000 - 0x000007fefc668000 	C:\windows\system32\CRYPTSP.dll
0x000007fefc080000 - 0x000007fefc0c7000 	C:\windows\system32\rsaenh.dll
0x000007fefcad0000 - 0x000007fefcadf000 	C:\windows\system32\CRYPTBASE.dll
0x000007fefabf0000 - 0x000007fefac01000 	C:\windows\system32\dhcpcsvc6.DLL
0x000007fefaab0000 - 0x000007fefaac8000 	C:\windows\system32\dhcpcsvc.DLL
0x0000000180000000 - 0x0000000180281000 	C:\Program Files\Apache Software Foundation\Tomcat 9.0_Tomcat9.0.16\bin\tcnative-1.dll
0x000007fefce00000 - 0x000007fefcf6d000 	C:\windows\system32\CRYPT32.dll
0x000007fefcc80000 - 0x000007fefcc8f000 	C:\windows\system32\MSASN1.dll
0x000007fefc590000 - 0x000007fefc5b2000 	C:\windows\system32\bcrypt.dll
0x000007fefc030000 - 0x000007fefc07c000 	C:\windows\system32\bcryptprimitives.dll
0x00000000709b0000 - 0x00000000709d4000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\sunec.dll
0x00000000709a0000 - 0x00000000709ab000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\sunmscapi.dll
0x000000006f380000 - 0x000000006f518000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\awt.dll
0x000007fefd850000 - 0x000007fefd92a000 	C:\windows\system32\OLEAUT32.dll
0x000007fefca70000 - 0x000007fefcac7000 	C:\windows\system32\apphelp.dll
0x0000000070d90000 - 0x0000000070dc1000 	C:\Program Files\Java\jdk1.8.0_152\jre\bin\jpeg.dll
0x000007feface0000 - 0x000007fefacf8000 	C:\windows\system32\DWMAPI.DLL
0x000007fef98e0000 - 0x000007fef9a05000 	C:\windows\system32\DBGHELP.DLL