Bug 62023

Summary: Tomcat crashes when SSLprotocol value is defined
Product: Tomcat 8 Reporter: joe.fletcher
Component: ConnectorsAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal    
Priority: P2    
Version: 8.5.23   
Target Milestone: ----   
Hardware: HP   
OS: Linux   
Attachments: Java error log file

Description joe.fletcher 2018-01-19 13:20:36 UTC
Created attachment 35687 [details]
Java error log file

Tomcat 8.5.23 with Oracle jdk 1.8.0_152 with tomcat native library 1.2.16, openssl 1.0.2l running on RHEL7.3

Any attempt to define a value for SSLProtocol causes crash on startup.


Using CATALINA_BASE:   /home/tomcat
Using CATALINA_HOME:   /home/tomcat
Using CATALINA_TMPDIR: /home/tomcat/temp
Using JRE_HOME:        /usr/java/latest
Using CLASSPATH:       /home/tomcat/bin/bootstrap.jar:/home/tomcat/bin/tomcat-juli.jar
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f164d78977a, pid=16600, tid=0x00007f166c39c700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_152-b16) (build 1.8.0_152-b16)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libtcnative-1.so.0.2.16+0x1d77a]  Java_org_apache_tomcat_jni_SSLContext_free+0xa
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/apache-tomcat-8.5.23/conf/hs_err_pid16600.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/home/tomcat/conf/../bin/catalina.sh: line 572: 16600 Aborted                 (core dumped) "/usr/java/latest/bin/java" -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath "/home/tomcat/bin/bootstrap.jar:/home/tomcat/bin/tomcat-juli.jar" -Dcatalina.base="/home/tomcat" -Dcatalina.home="/home/tomcat" -Djava.io.tmpdir="/home/tomcat/temp" org.apache.catalina.startup.Bootstrap configtest


Config file stanza reads thus:

  <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
                maxThreads="150"
                scheme="https"
                secure="true"
                SSLEnabled="true"
                SSLProtocol="+TLSv1"
                >

Any variation of the SSLProtocol definition produces the same result. Remove it and the instance starts without issue.

Sort of reminiscent of 52714. hs_err output file attached.
Comment 1 Remy Maucherat 2018-01-19 13:35:54 UTC
I have no idea why it crashes, so maybe this can stay open for now, but:
- Please read the documentation on SSL config
- The SSLProtocol attribute is a JSSE attribute and it doesn't do what you think it does
Comment 2 Mark Thomas 2018-01-19 13:43:09 UTC
Tested with latest 8.5.x.

That config snippet doesn't crash for me. Tomcat won't even start until the closing '>' is changed to '/>'. Then I get the expected warning about missing SSLCertificateFile

If you can reproduce this with the latest 8.5.x code please feel free to re-open. You'll need to provide the exact steps to reproduce this from a clean install.
Comment 3 joe.fletcher 2018-01-19 14:04:43 UTC
The  full connector config block is thus:

    <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
                maxThreads="150"
                scheme="https"
                secure="true"
                SSLEnabled="true"
                protocols="+TLSv1"
                >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
            <Certificate certificateKeyFile="/home/tomcat/conf/certs2/wild.wherever.com.key"
                         certificateFile="/home/tomcat/conf/certs2/wild.wherever.com.crt"
                         certificateChainFile="/home/tomcat/conf/certs2/intermediate.crt"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>


Note that in this instance I've tried the "protocols" attribute since apparently SSLProtocol isn't correct for this style of connector.
Comment 4 joe.fletcher 2018-01-19 14:28:46 UTC
@Remy/Mark:

Perhaps this is a documentation issue.

https://tomcat.apache.org/tomcat-8.0-doc/config/http.html#SSL_Support
https://tomcat.apache.org/tomcat-8.0-doc/ssl-howto.html

The ssl-howto doc gives this example.

The APR connector uses different attributes for many SSL settings, particularly keys and certificates. An example of an APR configuration is:

<!-- Define a SSL Coyote HTTP/1.1 Connector on port 8443 -->
<Connector
           protocol="org.apache.coyote.http11.Http11AprProtocol"
           port="8443" maxThreads="200"
           scheme="https" secure="true" SSLEnabled="true"
           SSLCertificateFile="/usr/local/ssl/server.crt"
           SSLCertificateKeyFile="/usr/local/ssl/server.pem"
           SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"/>

As stated, the config works fine provided I remove the SSLProtocol option.
Comment 5 Remy Maucherat 2018-01-19 14:59:19 UTC
Please do not reopen the report, this works for me too. About the doc, it is correct, I made a mistake about "The SSLProtocol attribute is a JSSE attribute and it doesn't do what you think it does", it's actually the "sslProtocol" attribute that is JSSE only, not "SSLProtocol" and it's confusing (and deprecated).
Comment 6 Mark Thomas 2018-01-19 17:20:59 UTC
Re-opening because I can reproduce this locally now the full configuration has been provided. The configuration may be invalid but we really should be displaying a useful error message rather than crashing. I might not get to this until next week.
Comment 7 Coty Sutherland 2018-01-22 15:10:03 UTC
I can reproduce this by adding any SSLHostConfig attribute to the Connector (namely `SSLVerifyClient="optional"`). The problem here is that when using APR and specifying Connector configuration which creates two '_default_' SSLHostConfig objects, tomcat-native crashes without providing any indication of what happened. That is not a great user experience :( This happens because org.apache.tomcat.util.net.AprEndpoint.releaseSSLContext() tries to release a zero context that tomcat-native asserts is non-zero (see tomcat-native native/src/sslcontext.c:363). After doing some tracing, it looks like AprEndpoint's releaseSSLContext method calls sslHostConfig.getOpenSslContext() and sslHostConfig.getOpenSslConfContext() which always seem to return 0. Is that correct behavior? It seems a bit buggy to me.

I was able to correct this behavior by adding two zero checks to the releaseSSLContext method in AprEndpoint:

 629     @Override
 630     protected void releaseSSLContext(SSLHostConfig sslHostConfig) {
 631         Long ctx = sslHostConfig.getOpenSslContext();
 632         if (ctx != null && ctx != 0) {
 633             SSLContext.free(ctx.longValue());
 634             sslHostConfig.setOpenSslContext(null);
 635         }
 636         Long cctx = sslHostConfig.getOpenSslConfContext();
 637         if (cctx != null && cctx != 0) {
 638             SSLConf.free(cctx.longValue());
 639             sslHostConfig.setOpenSslConfContext(null);
 640         }
 641     }

This causes the correct behavior to occur (at least it does the same thing as NioEndpoint and prints the endpoint.duplicateSslHostName message). I was going to push this, but given that I'm not sure how this OpenSSLContext stuff is supposed to work and the lack of javadocs in the new classes, I thought it best to get someone's opinion first.
Comment 8 Remy Maucherat 2018-01-22 15:18:47 UTC
Yes, something is wrong as there are null checks, yet, the sslHostConfig.getOpenSslContext() (same for the conf) return value cannot be null (the default is Long 0). Hummm, I would standardize on null as the new default value, so that if there's a problem it'll likely cause a NPE rather than a crash.
Comment 9 Rainer Jung 2018-01-22 15:39:40 UTC
(In reply to Coty Sutherland from comment #7)
> I can reproduce this by adding any SSLHostConfig attribute to the Connector
> (namely `SSLVerifyClient="optional"`). The problem here is that when using
> APR and specifying Connector configuration which creates two '_default_'
> SSLHostConfig objects, tomcat-native crashes without providing any
> indication of what happened. That is not a great user experience :( This
> happens because org.apache.tomcat.util.net.AprEndpoint.releaseSSLContext()
> tries to release a zero context that tomcat-native asserts is non-zero (see
> tomcat-native native/src/sslcontext.c:363). After doing some tracing, it
> looks like AprEndpoint's releaseSSLContext method calls
> sslHostConfig.getOpenSslContext() and sslHostConfig.getOpenSslConfContext()
> which always seem to return 0. Is that correct behavior? It seems a bit
> buggy to me.
> 
> I was able to correct this behavior by adding two zero checks to the
> releaseSSLContext method in AprEndpoint:
> 
>  629     @Override
>  630     protected void releaseSSLContext(SSLHostConfig sslHostConfig) {
>  631         Long ctx = sslHostConfig.getOpenSslContext();
>  632         if (ctx != null && ctx != 0) {
>  633             SSLContext.free(ctx.longValue());
>  634             sslHostConfig.setOpenSslContext(null);
>  635         }
>  636         Long cctx = sslHostConfig.getOpenSslConfContext();
>  637         if (cctx != null && cctx != 0) {
>  638             SSLConf.free(cctx.longValue());
>  639             sslHostConfig.setOpenSslConfContext(null);
>  640         }
>  641     }
> 
> This causes the correct behavior to occur (at least it does the same thing
> as NioEndpoint and prints the endpoint.duplicateSslHostName message). I was
> going to push this, but given that I'm not sure how this OpenSSLContext
> stuff is supposed to work and the lack of javadocs in the new classes, I
> thought it best to get someone's opinion first.

Yes, the aded check against 0 instead of only checking against null looks OK. The native pointer here is wrapper by a Long and the native null pointer would be a (Long)0.

Regards,

Rainer
Comment 10 joe.fletcher 2018-01-22 15:46:34 UTC
Might this be in any way related to problems creating a JSSE configuration?
I ran into what looked like https://bz.apache.org/bugzilla/show_bug.cgi?id=59910 when attempting to use a keystore based on a CA wildcard certificate.
It seems to insist on using an alias of "tomcat" but even with a single entry in the keystore with that alias it complains there is no valid certificate.
Comment 11 Coty Sutherland 2018-01-23 13:22:09 UTC
Fixed in:
- trunk for 9.0.5 onwards
- 8.5.x for 8.5.28 onwards

Thanks for the report!