Bug 63524

Summary: Private key must be accompanied by certificate chain
Product: Tomcat 8 Reporter: Arnaud Kleinveld <arnaud.kleinveld>
Component: ConnectorsAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal    
Priority: P2    
Version: 8.5.40   
Target Milestone: ----   
Hardware: PC   
OS: Linux   

Description Arnaud Kleinveld 2019-06-22 20:21:42 UTC
Upgrade to 8.5.40 broke the SSL connector. Connecting at port 8443 causes a connection time out. Catalina.out is reporting that is fails to initialise the connector at port 8443 as follows:

<snip>
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: Private key must be accompanied by certificate chain
        at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:404)
        at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:368)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1105)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
        ... 13 more
Caused by: java.lang.IllegalArgumentException: Private key must be accompanied by certificate chain
        at java.security.KeyStore.setKeyEntry(KeyStore.java:1136)
        at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:313)
        at org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers(OpenSSLUtil.java:105)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:239)
        at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:402)
        ... 18 more

Tomcat is configured to serve two domain names on two connectors each on a different ip address with the following configuration:

    <Connector port="8443" address="172.30.0.186" protocol="org.apache.coyote.http11.Http11AprProtocol" 
               maxThreads="200" SSLEnabled="true" 
               scheme="https" secure="true" SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
               SSLCertificateFile="/etc/pki/tls/certs/domain1.crt"
               SSLCACertificateFile="/etc/pki/tls/certs/comodo-ca-bundle.crt"
               SSLCertificateKeyFile="/etc/pki/tls/private/domain1.key" />

    <Connector port="8443" address="172.30.0.94" protocol="org.apache.coyote.http11.Http11AprProtocol" 
               maxThreads="200" SSLEnabled="true" 
               scheme="https" secure="true" SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
               SSLCertificateFile="/etc/pki/tls/certs/domain2.crt"
               SSLCACertificateFile="/etc/pki/tls/certs/lets-encrypt-x1-cross-signed.pem"
               SSLCertificateKeyFile="/etc/pki/tls/private/domain2.key" />

The above configuration was working fine until Tomcat upgrade to the latest AWS provided version which is 8.5.40 built May 2 2019 18:02:51 UTC
Comment 1 Arnaud Kleinveld 2019-06-23 00:09:18 UTC
The domain1 configuration is also in use by a httpd server which is working fine. Upgraded lets-encrypt-x1-cross-signed.pem to lets-encrypt-x3-cross-signed.pem for domain2, unfortunately that didn't make a difference.
Comment 2 Mark Thomas 2019-06-26 17:59:35 UTC
That sounds very much like bug 62526. That should be fixed in 8.5.40 though.

Are you using AWS Elastic Beanstalk? If not can you provide more environment details please. We may need to re-create this on AWS so we'll need to know exactly which service.
Comment 3 Mark Thomas 2019-06-26 19:01:28 UTC
Looking at this a bit more I haven't been able to reproduce it yet. I suspect it is related to the cert files being used. Is it possible for you to create a set of test files that reproduces the issue?
Comment 4 Mark Thomas 2019-06-26 21:06:38 UTC
Success! From a certain point of view. I have been able to recreate this. You will see this error if you certs are in DER rather than PEM format.

OpenSSL can handle DER quite happily but the code we added to enable you to switch seamlessly between OpenSSL and JSSE only works with PEM.

As a minimum, we should be able to get DER certs working again with OpenSSL. Getting them working with JSSE as well might be more of a challenge. Meanwhile conversion to PEM format should get everything working for you.
Comment 5 Arnaud Kleinveld 2019-06-26 23:37:10 UTC
Hi Mark, thank you for your quick and comprehensive replies. I am not using Beanstalk and have instead manually configured Tomcat. As far as I know my certificates are in PEM format because I can read them as text files. I Googled for a method to understand which type it is. According the article found DER certificates should be in binary format.

Another thing I forgot to mention is that I have another server running httpd using the same certificates and keys and that server works fine. Maybe that helps.
Comment 6 Arnaud Kleinveld 2019-06-26 23:41:37 UTC
(In reply to Mark Thomas from comment #4)
> Success! From a certain point of view. I have been able to recreate this.
> You will see this error if you certs are in DER rather than PEM format.
> 
> OpenSSL can handle DER quite happily but the code we added to enable you to
> switch seamlessly between OpenSSL and JSSE only works with PEM.
> 
> As a minimum, we should be able to get DER certs working again with OpenSSL.
> Getting them working with JSSE as well might be more of a challenge.
> Meanwhile conversion to PEM format should get everything working for you.

I noticed a message in Tomcat catalina.out log regarding this that may help: 

"The certificate [/etc/pki/tls/certs/domain2.crt] or its private key [/etc/pki/tls/private/domain2.key] could not be processed using a JSSE key manager and will be given directly to OpenSSL


On another note I also use Tomcat APR which is configured to use OpenSSL.
Comment 7 Mark Thomas 2019-06-27 16:23:58 UTC
There should be lines of headers / footers in each file starting "---". Can you list all the headers and footers present in each file please (this should help to ID the format being used).
Comment 8 Arnaud Kleinveld 2019-06-27 18:07:47 UTC
(In reply to Mark Thomas from comment #7)
> There should be lines of headers / footers in each file starting "---". Can
> you list all the headers and footers present in each file please (this
> should help to ID the format being used).

domain2.crt
-----BEGIN CERTIFICATE-----
<random chars>
-----END CERTIFICATE-----

domain2.key
-----BEGIN RSA PRIVATE KEY-----
<random chars>
-----END RSA PRIVATE KEY-----
Comment 9 Mark Thomas 2019-06-27 18:53:06 UTC
Thanks. That looks like a PEM encoded PKCS#1 key and a PEM encoded X509 cert.

When I start 8.5.40 with those I don't see the error you see. I've tested with the oldest and latest versions of OpenSSL.

I think we are back to me asking you to provide a set of test files (key and cert(s)) that reproduce the issue.
Comment 10 Christopher Schultz 2019-06-28 01:21:45 UTC
I realize that this conversation is headed in another direction, but...

(In reply to Mark Thomas from comment #4)
> OpenSSL can handle DER quite happily but the code we added to enable you to
> switch seamlessly between OpenSSL and JSSE only works with PEM.

The hard part is detecting the DER file, not reading it. JSSE will happily read a DER file in the same way it reads a PEM-encoded file:

FileInputStream fis = new FileInputStream("certificate.der");
CertificateFactory cf = CertificateFactory.getInstance("X.509");
Certificate cert = cf.generateCertificate(fis);

This will work either raw DER or PEM-encoded DER files. There are no hard and fast rules for reading multiple certificates in DER format so I think that DER files would have to be either single-cert only or we'd need to do some digging-around to see how other software handles multiple certs without PEM encoding.
Comment 11 Mark Thomas 2019-06-28 09:31:47 UTC
I've found various ways to trigger this error but I am not confident I have found the way the Arnaud is triggering the error.

Moving this to NEEDINFO until we get a set of test keys/certs that reproduce this.
Comment 12 Arnaud Kleinveld 2019-07-05 06:21:28 UTC
Hi, sorry for my late reply. I have gone through various options but I don't see how I can reproduce this error. The Apache httpd server is using the same certificates without any issues. Is there some kind of verbose logging I can filter on Tomcat that will help you?
Comment 13 Mark Thomas 2019-07-05 07:20:11 UTC
There isn't much in the way of logging to enable that would help here. What we really need is a set of keys/certs to reproduce the issue.

The simple solution (send me the keys/certs you are having the issue with) isn't really an option unless you really, really trust me since it means handing over your current private key(s). Not something I'd recommend.

Are you able to recreate the issue on a test server, revoke the cert(s) and then send me the files?

Another option would be to create a set of problematic keys/certs from scratch with OpenSSL and then add the steps to re-create those test files to this issue.
Comment 14 Arnaud Kleinveld 2019-07-05 08:19:31 UTC
Perhaps I can send to your email if you have a personal public key.
Comment 15 Mark Thomas 2019-07-05 11:53:28 UTC
My public keys for markt@apache.org are listed here:
http://people.apache.org/keys/committer/

The first one (A9C5 ...) is my preferred one.
Comment 16 Mark Thomas 2019-07-12 10:08:10 UTC
Thanks. I have received the reproduction information and I can reproduce this issue. I'm looking into what is going wrong now.
Comment 17 Mark Thomas 2019-07-12 15:15:06 UTC
There are two separate issues here.

The first is that the mechanism we are using to translate keys and certs to a common format internally is stricter than OpenSSL and requires a valid certificate chain. I have a patch that allows fall-back to direct OpenSSL configuration in this case. Alternatively, the issue can be worked-around by installed the cert chain. In this instance it is the "Sectigo RSA DV Bundle" from https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA01N000000rfBO

The second issue is that the mechanism we are using to translate keys and certificates to a common format doesn't support PKCS#1. Annoyingly, everything we need is in the JRE but in the non-public sun.security.util package. I have a patch to add PKCS#1 support as well.

I need to tidy the patches up, fill in the i18n that I skipped over and then I'll be in a position to commit this.
Comment 18 Mark Thomas 2019-07-12 15:56:34 UTC
Fixed in:
- master for 9.0.23 onwards
- 8.5.x for 8.5.44 onwards

The fix isn't needed in 7.0.x since the JSSE and OpenSSL configurations remain separate in 7.0.x
Comment 19 Arnaud Kleinveld 2019-07-13 01:27:41 UTC
Great and thank you for your excellent support. Looking forward to AWS update announcement. This may take a while I guess. Do I understand correctly that in the meantime I can solve the problem by installing the named bundle and using certs in another format than pkcs#?
Comment 20 Mark Thomas 2019-07-13 12:55:14 UTC
Yes. For the Sertigo key/cert use certificateChainFile="/path/to/dv/bundle".

For the letsencrypt key, if you convert it from PKCS#1 to PKCS#8 (opensssl can do this) that should be OK as well.
Comment 21 Arnaud Kleinveld 2019-07-28 15:53:31 UTC
(In reply to Mark Thomas from comment #20)
> Yes. For the Sertigo key/cert use certificateChainFile="/path/to/dv/bundle".
> 
> For the letsencrypt key, if you convert it from PKCS#1 to PKCS#8 (opensssl
> can do this) that should be OK as well.

Because the tomcat server is using APR I configured the server with SSLCACertificateFile. Is this correct or should the configuration be changed, to NIO2 I just read?
Comment 22 Mark Thomas 2019-07-28 18:39:19 UTC
No need to switch from APR/Native to NIO2.