Bug 63500 - Core dump using APR tomcat native with certificateRevocationListFile
Summary: Core dump using APR tomcat native with certificateRevocationListFile
Alias: None
Product: Tomcat Native
Classification: Unclassified
Component: Library (show other bugs)
Version: 1.2.21
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2019-06-13 15:18 UTC by Bruno
Modified: 2019-07-23 16:23 UTC (History)
0 users

Error report generated by the JVM (122.92 KB, text/plain)
2019-06-13 21:58 UTC, Bruno
Native stack trace from core dump of same process (6.29 KB, text/plain)
2019-06-13 22:07 UTC, Bruno
Native stack trace from core dump of same process with debuginfo (12.17 KB, text/plain)
2019-06-13 22:42 UTC, Bruno

Note You need to log in before you can comment on or make changes to this bug.
Description Bruno 2019-06-13 15:18:52 UTC
I'm using tomcat 9.0.21 and tomcat-native 1.2.21 on top of CentOS 7.6.1810.

Current issue is this one from catalina.out:
13-Jun-2019 10:43:32.132 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.4.8].
13-Jun-2019 10:43:32.135 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
13-Jun-2019 10:43:32.135 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
13-Jun-2019 10:43:32.141 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.0.2k-fips  26 Jan 2017]
13-Jun-2019 10:43:32.518 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-apr-2007"]
13-Jun-2019 10:43:32.545 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-apr-2008"]
# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x00007f989117c860, pid=23986, tid=0x00007f9910185700
# JRE version: OpenJDK Runtime Environment (8.0_191-b12) (build 1.8.0_191-b12)
# Java VM: OpenJDK 64-Bit Server VM (25.191-b12 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libcrypto.so.10+0x168860]  X509_STORE_set_flags+0x0
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# An error report file with more information is saved as:
# /opt/temp/java_error23986.log
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp

The crash log shows

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.tomcat.jni.SSLContext.setCARevocation(JLjava/lang/String;Ljava/lang/String;)Z+0
j  org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(Lorg/apache/tomcat/util/net/SSLHostConfigCertificate;)V+77
j  org.apache.tomcat.util.net.openssl.OpenSSLContext.init([Ljavax/net/ssl/KeyManager;[Ljavax/net/ssl/TrustManager;Ljava/security/SecureRandom;)V+191
j  org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(Ljava/util/List;)Lorg/apache/tomcat/util/net/SSLContext;+16
j  org.apache.tomcat.util.net.AprEndpoint.createSSLContext(Lorg/apache/tomcat/util/net/SSLHostConfig;)V+81

Which is the code https://github.com/apache/tomcat-native/blob/7a20a500587dc92944dee05cee5b388ae8443842/native/src/sslcontext.c#L640

Connector configuration:
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" SSLRandomSeed="builtin" />
        protocols="all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.3"
          certificateKeyPassword="abc" />

When I remove the line setting the revogation list, "certificateRevocationListFile=\"/certs/server/x509_trusted.crl\"", it just works. Same PEM file works in old tomcat-7.0.69 using also APR and tomcat native.
Tried versions back until 9.0.17 with same issue. With 9.0.16 and 9.014 I have different issue.

On tomcat 9.0.14 same setup worked but on shutdown I had a core dump. With tomcat 9.0.16 same setup worked but on shutdown I had an even weird core dump. Starting tomcat 9.0.17 it stopped starting. I noticed that tomcat native changed from 1.2.19 to 1.2.21 when that happen.

Tested also building a tomcat-native with latest openssl and latest apr as static linked in with same issue. So I assume it's not some weird library issue but something else.

Happy to provide more data or even do other tests.
Comment 1 Christopher Schultz 2019-06-13 17:57:21 UTC
Looks like this fails on startup. Can you enable core dumps, re-try, and get a native stack trace of the error?
Comment 2 Bruno 2019-06-13 21:58:32 UTC
Created attachment 36623 [details]
Error report generated by the JVM
Comment 3 Bruno 2019-06-13 22:07:40 UTC
Created attachment 36624 [details]
Native stack trace from core dump of same process

gdb java core
bt full
Comment 4 Bruno 2019-06-13 22:09:42 UTC
It does fail right at the beginning.

Attached the traces.
Comment 5 Bruno 2019-06-13 22:42:55 UTC
Created attachment 36625 [details]
Native stack trace from core dump of same process with debuginfo

Installed the debug info and retrieved new native trace.
Comment 6 Mark Thomas 2019-06-17 20:24:34 UTC
The stack trace suggests the fix for bug 62094 was incorrect.

Looking at the trace, the proposed patch and the commit, I think the patch was not applied correctly.
Comment 7 Mark Thomas 2019-06-18 21:02:30 UTC
Correcting product
Comment 8 Mark Thomas 2019-06-19 17:21:25 UTC
I've dug into this some more. I see why the applied patch for bug 62094 differs from the proposed patch but the change made was incomplete. That was the cause of the crash observed here.

Further testing found crashes on invalid CRL files and on shutdown. These have also been fixed.

I have tested this with a CRL and both a revoked and non-revoked cert and everything appears to as expected.
Comment 9 Bruno 2019-06-20 17:20:11 UTC
Thank you for the investigation and fix.

What is the release cycle for tomcat native and tomcat 9?
Comment 10 Mark Thomas 2019-06-20 17:52:09 UTC
Current plans are:

- 9.0.x early July
- Native 1.2.22 before then so it can be packaged in the next 9.0.x release

You can follow progress on the dev@ list.
Comment 11 Bruno 2019-07-23 16:23:51 UTC
Using the fix with tomcat 9.0.22 and tomcat-native 1.2.23 and the server doesn't core dump anymore when I add CRL to SSLHostConfig using configuration certificateRevocationListFile in format
-----BEGIN X509 CRL-----

But have a new issue, I'm unable to use valid X509 client certs. I get the message from curl

curl: (35) error:1401E418:SSL routines:CONNECT_CR_FINISHED:tlsv1 alert unknown ca

If I remove the CRL config client certs work as expected.

Is there a way to understand what is the issue? Was unable to find a way to enable SSL logging being OpenSSL.