Bug 67666 - TLSCertificateReloadListener does not detect all certificates to reload
Summary: TLSCertificateReloadListener does not detect all certificates to reload
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 9.0.81
Hardware: All All
: P2 major (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-10 18:40 UTC by Michael Osipov
Modified: 2023-10-25 10:57 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Osipov 2023-10-10 18:40:41 UTC
Tested with 9.0.82-dev, but I guess this happens on all versions

Created one certificate:
openssl req -x509 -newkey rsa:4096 -keyout key.crt -out cert.crt -sha256 -days 5 -passout file:key-password
openssl pkcs12 -export -in cert.crt -inkey key.crt -out keystore.p12 -name "localhost" -passin file:key-password -passout file:keystore-password

Declared in server.xml:
<Listener className="org.apache.catalina.security.TLSCertificateReloadListener" checkPeriod="120" daysBefore="360" />

and 

    <Connector port="20001" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true"
               maxParameterCount="1000"
               sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation">
        <SSLHostConfig>
            <Certificate certificateKeyFile="conf/certs-localhost/key.crt"
                         certificateFile="conf/certs-localhost/cert.crt"
                         certificateChainFile="conf/cacerts.crt"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Connector port="20002" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true"
               maxParameterCount="1000"
               sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation">
        <SSLHostConfig>
            <Certificate certificateKeystoreFile="conf/certs-localhost/keystore.p12"
                         certificateKeyAlias="localhost"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Connector port="20003" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true"
               maxParameterCount="1000"
               sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation">
        <SSLHostConfig>
            <Certificate certificateKeyFile="conf/certs-localhost/key.crt"
                         certificateFile="conf/certs-localhost/cert.crt"
                         certificateChainFile="conf/cacerts.crt"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Connector port="20004" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true"
               maxParameterCount="1000"
               sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation">
        <SSLHostConfig>
            <Certificate certificateKeystoreFile="conf/certs-localhost/keystore.p12"
                         certificateKeyAlias="localhost"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Connector port="30001" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true"
               maxParameterCount="1000">
        <SSLHostConfig>
            <Certificate certificateKeyFile="conf/certs-localhost/key.crt"
                         certificateFile="conf/certs-localhost/cert.crt"
                         certificateChainFile="conf/cacerts.crt"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Connector port="30002" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true"
               maxParameterCount="1000">
        <SSLHostConfig>
            <Certificate certificateKeystoreFile="conf/certs-localhost/keystore.p12"
                         certificateKeyAlias="localhost"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

Starting Tomcat:
10-Oct-2023 20:21:25.310 INFORMATION [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-20001"]
10-Oct-2023 20:21:25.802 SCHWERWIEGEND [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["https-jsse-nio-20001"]]
        org.apache.catalina.LifecycleException: Protocol handler initialization failed
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1011)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:554)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1039)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:127)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
                at java.base/java.lang.reflect.Method.invoke(Method.java:578)
                at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:307)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:477)
        Caused by: java.lang.IllegalArgumentException: PBE parameter parsing error: expecting the object identifier for AES cipher
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:107)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:236)
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1326)
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1339)
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:654)
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:75)
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1009)
                ... 11 more
        Caused by: java.io.IOException: PBE parameter parsing error: expecting the object identifier for AES cipher
                at java.base/com.sun.crypto.provider.PBES2Parameters.parseES(PBES2Parameters.java:324)
                at java.base/com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:240)
                at java.base/java.security.AlgorithmParameters.init(AlgorithmParameters.java:311)
                at java.base/sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:149)
                at java.base/sun.security.x509.AlgorithmId.<init>(AlgorithmId.java:131)
                at java.base/sun.security.x509.AlgorithmId.parse(AlgorithmId.java:416)
                at java.base/javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:105)
                at org.apache.tomcat.util.net.jsse.PEMFile$Part.toPrivateKey(PEMFile.java:245)
                at org.apache.tomcat.util.net.jsse.PEMFile.<init>(PEMFile.java:178)
                at org.apache.tomcat.util.net.jsse.PEMFile.<init>(PEMFile.java:107)
                at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:355)
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:268)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:105)
                ... 18 more
10-Oct-2023 20:21:25.806 INFORMATION [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-20002"]
10-Oct-2023 20:21:25.953 INFORMATION [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-jsse-nio-20002], TLS virtual host [_default_], certificate type [RSA] configured from keystore [conf/certs-localhost/keystore.p12] using alias [localhost] with trust store [null]
10-Oct-2023 20:21:25.986 INFORMATION [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-20003"]
10-Oct-2023 20:21:26.013 INFORMATION [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [conf/certs-localhost/cert.crt] or its private key [conf/certs-localhost/key.crt] could not be processed using a JSSE key manager and will be given directly to OpenSSL
10-Oct-2023 20:21:26.117 INFORMATION [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-nio-20003], TLS virtual host [_default_], certificate type [RSA] configured from key [conf/certs-localhost/key.crt], certificate [conf/certs-localhost/cert.crt] and certificate chain [conf/cacerts.crt] with trust store [null]
10-Oct-2023 20:21:26.119 INFORMATION [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-20004"]
10-Oct-2023 20:21:26.141 INFORMATION [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-nio-20004], TLS virtual host [_default_], certificate type [RSA] configured from keystore [conf/certs-localhost/keystore.p12] using alias [localhost] with trust store [null]
10-Oct-2023 20:21:26.143 INFORMATION [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-apr-30001"]
10-Oct-2023 20:21:26.157 INFORMATION [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [conf/certs-localhost/cert.crt] or its private key [conf/certs-localhost/key.crt] could not be processed using a JSSE key manager and will be given directly to OpenSSL
10-Oct-2023 20:21:26.209 INFORMATION [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [conf/certs-localhost/cert.crt] or its private key [conf/certs-localhost/key.crt] could not be processed using a JSSE key manager and will be given directly to OpenSSL
10-Oct-2023 20:21:26.210 INFORMATION [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-apr-30001], TLS virtual host [_default_], certificate type [RSA] configured from key [conf/certs-localhost/key.crt], certificate [conf/certs-localhost/cert.crt] and certificate chain [conf/cacerts.crt] with trust store [null]
10-Oct-2023 20:21:26.212 INFORMATION [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-apr-30002"]
10-Oct-2023 20:21:26.248 INFORMATION [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-apr-30002], TLS virtual host [_default_], certificate type [RSA] configured from keystore [conf/certs-localhost/keystore.p12] using alias [localhost] with trust store [null]
10-Oct-2023 20:21:26.250 INFORMATION [main] org.apache.catalina.startup.Catalina.load Server initialization in [2178] milliseconds

We will ignore 20001 for now because Java does not support DES-encrypted private keys, only AES.

Waiting for the listener:
10-Oct-2023 20:35:11.441 INFORMATION [Catalina-utility-1] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-jsse-nio-20002], TLS virtual host [_default_], certificate type [RSA] configured from keystore [conf/certs-localhost/keystore.p12] using alias [localhost] with trust store [null]
10-Oct-2023 20:35:11.442 INFORMATION [Catalina-utility-1] org.apache.catalina.security.TLSCertificateReloadListener.checkCertificatesForRenewal [Connector["https-jsse-nio-20002"]], TLS virtual host [_default_] reloaded TLS configuration
10-Oct-2023 20:35:11.442 WARNUNG [Catalina-utility-1] org.apache.catalina.security.TLSCertificateReloadListener.checkCertificatesForRenewal [Connector["https-jsse-nio-20002"]], TLS virtual host [_default_] with name [CN=localhost, OU=IN IT IN, O=Siemens, L=Berlin, ST=Berlin, C=DE] that expires on [2023-10-15T17:20:55Z] is overdue for renewal
10-Oct-2023 20:35:11.456 INFORMATION [Catalina-utility-1] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-nio-20004], TLS virtual host [_default_], certificate type [RSA] configured from keystore [conf/certs-localhost/keystore.p12] using alias [localhost] with trust store [null]
10-Oct-2023 20:35:11.457 INFORMATION [Catalina-utility-1] org.apache.catalina.security.TLSCertificateReloadListener.checkCertificatesForRenewal [Connector["https-openssl-nio-20004"]], TLS virtual host [_default_] reloaded TLS configuration
10-Oct-2023 20:35:11.457 WARNUNG [Catalina-utility-1] org.apache.catalina.security.TLSCertificateReloadListener.checkCertificatesForRenewal [Connector["https-openssl-nio-20004"]], TLS virtual host [_default_] with name [CN=localhost, OU=IN IT IN, O=Siemens, L=Berlin, ST=Berlin, C=DE] that expires on [2023-10-15T17:20:55Z] is overdue for renewal
10-Oct-2023 20:35:11.476 INFORMATION [Catalina-utility-1] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-apr-30002], TLS virtual host [_default_], certificate type [RSA] configured from keystore [conf/certs-localhost/keystore.p12] using alias [localhost] with trust store [null]
10-Oct-2023 20:35:11.476 INFORMATION [Catalina-utility-1] org.apache.catalina.security.TLSCertificateReloadListener.checkCertificatesForRenewal [Connector["https-openssl-apr-30002"]], TLS virtual host [_default_] reloaded TLS configuration
10-Oct-2023 20:35:11.477 WARNUNG [Catalina-utility-1] org.apache.catalina.security.TLSCertificateReloadListener.checkCertificatesForRenewal [Connector["https-openssl-apr-30002"]], TLS virtual host [_default_] with name [CN=localhost, OU=IN IT IN, O=Siemens, L=Berlin, ST=Berlin, C=DE] that expires on [2023-10-15T17:20:55Z] is overdue for renewal

Connectors for ports 20003 and 30001 do not appear because they use at some point "X509KeyManager x509KeyManager = certificate.getCertificateKeyManager();" returns null, thus "X509Certificate[] certificates = sslContext.getCertificateChain(alias);" is null and certificatesExpiringBefore() returns an empty set.

I assume that the code needs refinement, I guess that renew bots will create OpenSSL-style cert and key this listener won't be usable for people at the moment.
Comment 1 Michael Osipov 2023-10-11 09:30:42 UTC
Before reload:
$ for port in 20001 20002 20003 20004 30001 30002 ; do echo $port; echo -n Q | openssl s_client -connect localhost:$port 2>/dev/null | openssl x509 -noout -dates; done
20001
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT
20002
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT
20003
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT
20004
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT
30001
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT
30002
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT

After reload:
osipovmi@md2pcvtc MINGW64 ~
$ for port in 20001 20002 20003 20004 30001 30002 ; do echo $port; echo -n Q | openssl s_client -connect localhost:$port 2>/dev/null | openssl x509 -noout -dates; done
20001
notBefore=Oct 11 08:17:18 2023 GMT
notAfter=Oct 10 08:17:18 2024 GMT
20002
notBefore=Oct 11 08:17:18 2023 GMT
notAfter=Oct 10 08:17:18 2024 GMT
20003
notBefore=Oct 11 07:32:31 2023 GMT
notAfter=Oct 16 07:32:31 2023 GMT
20004
notBefore=Oct 11 08:17:18 2023 GMT
notAfter=Oct 10 08:17:18 2024 GMT
30001
notBefore=Oct 11 08:17:18 2023 GMT
notAfter=Oct 10 08:17:18 2024 GMT
30002
notBefore=Oct 11 08:17:18 2023 GMT
notAfter=Oct 10 08:17:18 2024 GMT

Very weird...I certain this needs more testing/anaylsis...
Comment 2 Mark Thomas 2023-10-24 21:53:35 UTC
The fix for BZ 67667 addressed most of the errors. The remaining issue has been fixed.

Reloading isn't going to work for PEM files that are passed directly to OpenSSL although with BZ 67667 those should be few and far between. And we should be able to add handling for them once the PEM format (primarily the KDF and cipher) are known. When reloading isn't going to work, a warning will be logged as there are a few management/monitoring features that won't work.

Fixed in:
- main for 11.0.0-M13 and onwards
- 10.1.x for 10.1.16 and onwards
- 9.0.x  for 9.0.83 and onwards
- 8.5.x for 8.5.96 and onwards
Comment 3 Mark Thomas 2023-10-24 21:56:00 UTC
Sorry, wrong bug reference. It was bug 67675.
Comment 4 Michael Osipov 2023-10-25 10:57:30 UTC
Just build Tomcat from bec7a51d7fc3fb913c755b258169d1816b77bea5. I can confirm that is works now.