Bug 65181 - Tomcat Native library with OpenSSL Engine private key loading
Summary: Tomcat Native library with OpenSSL Engine private key loading
Status: RESOLVED FIXED
Alias: None
Product: Tomcat Native
Classification: Unclassified
Component: Library (show other bugs)
Version: 1.2.26
Hardware: PC Linux
: P1 enhancement (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-12 20:14 UTC by Edin Hodzic
Modified: 2021-04-01 10:57 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Edin Hodzic 2021-03-12 20:14:17 UTC
I am new to Tomcat and to this mailing list. Looked far and wide for a solution to my problem, but couldn't find anything effective. I found other folks asking about similar issues. I then looked through the source and think I got a solution that I'd like to share as a patch.

The problem is this: Trying to use Apache Tomcat with an OpenSSL Engine that has proprietary private ECC key format fails. The private key file is not PEM, and only this specific OpenSSL Engine can load such a private ECC key. When the server.xml configuration includes reference to a proprietary format private ECC key, in a Service/Connector/SSLHostConfig/Certificate/certificateKeyFile, the run-time fails to initialize a new SSL context. As a result, TLS doesn't get established, connection fails.

I have tried Tomcat7, 9 and 10.

To illustrate the configuration in server.xml, it includes elements like these:

    <Listener
        className="org.apache.catalina.core.AprLifecycleListener"
        SSLEngine="MySslEngine" />

    <Service name="Catalina">

      <!-- Define an SSL Coyote HTTP/1.1 Connector on port 8443 -->
      <Connector
          protocol="org.apache.coyote.http11.Http11NioProtocol"
          port="8443"
          maxThreads="200"
          scheme="https"
          SSLEnabled="true">

        <SSLHostConfig
            caCertificateFile="my-keys/ca.pem"
            certificateVerification="require"
            certificateVerificationDepth="10">

          <Certificate
              certificateKeyFile="my-keys/server.key"
              certificateFile="my-keys/server.pem"
              certificateChainFile="my-keys/server-chain.pem"
              type="EC" />

        </SSLHostConfig>
      </Connector>

      <!-- ... -->

    </Service>

The logs may include lines like these:

05-Mar-2021 14:37:07.175 INFO [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [/opt/my-keys/server.pem] or its private key [/opt/my-keys/server.key] could not be processed using a JSSE key manager and will be given directly to OpenSSL                                                                                                                                                          
05-Mar-2021 14:37:07.176 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context                                                                                        
        java.lang.Exception: Unable to load certificate key /opt/my-keys/server.key (error:0909006C:PEM routines:get_name:no start line)                                                                            
                at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method)                                                                                                                                    
                at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379)                                                                                                        
                at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250)                                                                                                                  
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)                                                                                                                    
                at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:401)                                                                                                                    
                at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:367)                                                                                                                                
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1164)                                                                                                          
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1177)                                                                                                                      
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:574)                                                                                                                                
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82)                                                                                                              
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052)                                                                                                                        
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)                                                                                                                              
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:558)                                                                                                                  
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)                                                                                                                              
                at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)                                                                                                                    
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)                                                                                                                              
                at org.apache.catalina.startup.Catalina.load(Catalina.java:747)                                                                                                                                      
                at org.apache.catalina.startup.Catalina.load(Catalina.java:769)                                                                                                                                      
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                      
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                    
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                            
                at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                                                  
                at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)                                                                                                                                    
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)                                                                                                                                    


My understanding of the root cause is that Tomcat doesn't support a proprietary format of the private ECC key. It insists that the key be in PEM format, in a file or in a keystore.
What I needed was support for the "engine" key format. Similar to the feature of the "openssl digest" command in the following invocation:

openssl dgst                     \
        -sign my-keys/server.key \
        -keyform ENGINE          \
        -engine MySslEngine      \
        -out signature.bin       \
        my-input

When the key has the form "engine", the key is loaded using the ENGINE_load_private_key API (https://www.openssl.org/docs/man1.1.0/man3/ENGINE_load_private_key.html).

I have come up with a small change to the Tomcat Native library that resolves the problem for me. It is not as general as the "engine" key form in the openssl command line. The change below simply attempts to load the private key through the ENGINE_load_private_key if load_pem_key fails. Please consider the change as a patch to the Tomcat Native library:

--- tomcat-native-1.2.26-src/native/include/ssl_private.h 2020-12-10 09:09:19.000000000 -0800
+++ tomcat-native-1.2.26-src.changed/native/include/ssl_private.h 2021-03-08 15:35:56.516968836 -0800
@@ -51,6 +51,7 @@
  */
 #ifndef OPENSSL_NO_ENGINE
 #include <openssl/engine.h>
+extern ENGINE *tcn_ssl_engine;
 #endif
 
 #ifndef RAND_MAX
--- tomcat-native-1.2.26-src/native/src/sslcontext.c 2020-12-10 09:09:19.000000000 -0800
+++ tomcat-native-1.2.26-src.changed/native/src/sslcontext.c 2021-03-08 15:38:18.264651853 -0800
@@ -1034,7 +1034,13 @@
         }
     }
     else {
-        if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL) {
+        if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL
+#ifndef OPENSSL_NO_ENGINE
+            && tcn_ssl_engine &&
+            (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file,
+                                                    NULL, NULL)) == NULL
+#endif
+            ) {
             ERR_error_string(SSL_ERR_get(), err);
             tcn_Throw(e, "Unable to load certificate key %s (%s)",
                       key_file, err);
Comment 1 Edin Hodzic 2021-03-18 18:55:12 UTC
Changed Importance to P1 / Major as this blocks use of OpenSSL Engine with TPMs or HSMs, and OpenSSL Engine support is included in Tomcat Native library already.
Comment 2 Christopher Schultz 2021-03-19 20:54:34 UTC
Changing back to "enhancement" since that's what this is.

It may be of major importance to YOU but this is _not_ a bug.
Comment 3 Christopher Schultz 2021-03-19 21:17:31 UTC
A quick review:

>    ...
>    && tcn_ssl_engine &&
>    (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file,
>                                            NULL, NULL)) == NULL

This will attempt to use the tcn_ssl_engine pointer to load the private key, but only if the pointer is non-zero. If you mean NULL, you should explicitly compare the pointer to NULL.

Otherwise the patch looks reasonable to me.

I assume you've tested it to make sure it still works with a "standard" configuration?
Comment 4 Edin Hodzic 2021-03-19 21:39:34 UTC
Thanks for the updates, the review and the comments.

The NULL comparison I can add.

The change was tested with and without the SSL engine in the configuration.

Is there anything else I can do to get the patch included in the source?
Could I submit a pull request somehow?
Comment 5 mgrigorov 2021-03-23 14:23:35 UTC
(In reply to Edin Hodzic from comment #4)
> Could I submit a pull request somehow?

https://github.com/apache/tomcat-native
Comment 6 Mark Thomas 2021-03-25 20:32:30 UTC
Thanks for the patch. The fix will be in 1.2.27.

The Tomcat 10.0.x unit tests passed for the APR/Native connector with this patch applied.
Comment 7 Edin Hodzic 2021-03-29 16:39:40 UTC
Thanks everyone for the prompt action and the pending change.
Comment 8 Konstantin Kolinko 2021-03-30 16:46:02 UTC
Looking at commit, that implemented this feature for Tomcat Native 1.2.27,
1. There is a bug.

https://github.com/apache/tomcat-native/commit/69e884a96a308a2bfdd91c7de3a9b301838031c8

in native/src/sslcontext.c, lines 1039-1040

> -        if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL) {
> +        if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL
> +#ifndef OPENSSL_NO_ENGINE
> +                && tcn_ssl_engine != NULL &&
> +                (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file,
> +                                                        NULL, NULL)) == NULL
> +#endif
> +                ) {

I think that when one uses the default configuration (when there is no custom engine configured with SSLEngine attribute of AprLifecycleListener) and thus tcn_ssl_engine is NULL, the behaviour should remain unchanged.

As such, the code above has to be
"&& (tcn_ssl_engine == NULL || ENGINE_load_private_key() fails)".

With the current code when load_pem_key call fails and tcn_ssl_engine == NULL the error reporting block is skipped.


2. I am OK with the patch with the above bug fixed,
but for future improvements there are some thoughts:

1) I wonder, whether in case of a custom engine (tcn_ssl_engine != NULL) it would be better to try a "ENGINE_load_private_key" call first.

2) As "load_pem_key" is our own method (implemented in the same sslcontext.c file), I wonder whether it would be better to move the change there (and maybe rename that method).

3) I see that "ENGINE_load_private_key" method will be deprecated in OpenSSL 3.0 (master)
https://www.openssl.org/docs/man1.1.1/man3/ENGINE_load_private_key.html
https://www.openssl.org/docs/manmaster/man3/ENGINE_load_private_key.html

4) How about a "ENGINE_load_public_key" method? There is no need to call it?



STEPS TO REPRODUCE (for 1. bug)

Using release candidate of Tomcat 9.0.45:
1) Install Tomcat, e.g. unzip apache-tomcat-9.0.45-windows-x64.zip on Windows.
2) Edit the conf/server.xml file.
- ~line 68: Commend out the default HTTP connector on port 8080
- ~line 102: Uncomment a connector that uses OpenSSL for TLS
3) Create two files with some dummy content (e.g. "foo bar"):
conf/localhost-rsa-cert.pem
conf/localhost-rsa-key.pem
4) Start Tomcat.

ACTUAL BEHAVIOUR:

With Tomcat 9.0.45 and Tomcat Native 1.2.27 a startup failure is reported as follows:

[[[
30-Mar-2021 18:28:20.133 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context
	java.lang.Exception: Unable to load certificate [censored]\conf/localhost-rsa-cert.pem (error:0D06B08E:asn1 encoding routines:asn1_d2i_read_bio:not enough data)
		at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method)
		at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379)
		at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250)
		at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
		at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:397)
		at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:363)
		at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1204)
		at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1217)

]]]

EXPECTED BEHAVIOUR:

If I replace bin/tcnative-1.dll file with Tomcat Native 1.2.26 (e.g. the one from Tomcat 9.0.44), the error is reported as follows:

[[[
30-Mar-2021 18:30:02.551 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context
	java.lang.Exception: Unable to load certificate key [censored]\conf/localhost-rsa-key.pem (error:0909006C:PEM routines:get_name:no start line)
		at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method)
		at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379)
		at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250)
		at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
		at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:397)
		at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:363)
		at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1204)
		at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1217)
]]]

As loading the key in SSLContext::setCertificate (in native/src/sslcontext.c) happens earlier than loading the certificate, the expected error message is "Unable to load certificate key".

Seeing the "Unable to load certificate" message means that error handling for the former was skipped.
Comment 9 Edin Hodzic 2021-03-30 16:59:23 UTC
> As such, the code above has to be
"&& (tcn_ssl_engine == NULL || ENGINE_load_private_key() fails)".

I agree, my bad on the original patch. :/
Comment 10 Mark Thomas 2021-04-01 10:57:41 UTC
Fixed for 1.2.28 onwards.