Bug 52212

Summary: SSLProxyMachineCertificateFile key first causes segfault
Product: Apache httpd-2 Reporter: Keith Burdis <keith>
Component: mod_sslAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: normal Keywords: FixedInTrunk
Priority: P2    
Version: 2.2.21   
Target Milestone: ---   
Hardware: All   
OS: All   

Description Keith Burdis 2011-11-18 23:12:04 UTC
Sending a request via a mod_proxy reverse proxy to an https backend results in a segfault. This is a similar issue to bug 24030 in that the segfault also occurs in ssl_callback_proxy_cert calling the modssl_set_cert_info() macro:

   #define modssl_set_cert_info(info, cert, pkey) \
       *cert = info->x509; \
       X509_reference_inc(*cert); \
       *pkey = info->x_pkey->dec_pkey; \
       EVP_PKEY_reference_inc(*pkey)

except that instead of info->x_pkey being NULL, it is info->x_pkey->dec_pkey (the private key data) that is NULL.

(gdb) p *info                                                                                 $1 = {x509 = 0xcb2140, crl = 0x0, x_pkey = 0xcaf490, enc_cipher = {cipher = 0x0,                  iv = "?\"?\000\000\000\000\000?\t?\000\000\000\000"}, enc_len = 0, enc_data = 0x0,          references = 1}                                                                             (gdb) set print pretty on
(gdb) p info->x_pkey->dec_pkey
$5 = (EVP_PKEY *) 0x0
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000002a96f10e82 in CRYPTO_add_lock (pointer=0x8, amount=1, type=5,
    file=0x2a96fbdd7f "ssl_engine_kernel.c", line=1698) at cryptlib.c:630
^Z^Z/app/home/calrn13t/dl/openssl-1.0.0e/crypto/cryptlib.c:630:18583:beg:0x2a96f10e82
(gdb) bt
#0  0x0000002a96f10e82 in CRYPTO_add_lock (pointer=0x8, amount=1, type=5,
    file=0x2a96fbdd7f "ssl_engine_kernel.c", line=1698) at cryptlib.c:630
#1  0x0000002a96edd0a3 in ssl_callback_proxy_cert (ssl=0xcbc710, x509=0x7fbfffea28,
    pkey=0x7fbfffea30) at ssl_engine_kernel.c:1698
#2  0x0000002a96f09deb in ssl_do_client_cert_cb (s=0xcbc710, px509=0x7fbfffea28,
    ppkey=0x7fbfffea30) at s3_clnt.c:3048
#3  0x0000002a96f09e89 in ssl3_send_client_certificate (s=0xcbc710) at s3_clnt.c:2812
#4  0x0000002a96f0a378 in ssl3_connect (s=0xcbc710) at s3_clnt.c:373
...

I initially avoided the segfault by modifying the fix for bug 24030 to check x_pkey->dec_pkey as well:

--- a/httpd-2.2.21/modules/ssl/ssl_engine_init.c
+++ b/httpd-2.2.21/modules/ssl/ssl_engine_init.c
@@ -1006,7 +1006,7 @@ static void ssl_init_proxy_certs(server_rec *s,
     for (n = 0; n < ncerts; n++) {
         X509_INFO *inf = sk_X509_INFO_value(sk, n);

-        if (!inf->x509 || !inf->x_pkey) {
+        if (!inf->x509 || !inf->x_pkey || !inf->x_pkey->dec_pkey) {
             sk_X509_INFO_free(sk);
             ap_log_error(APLOG_MARK, APLOG_STARTUP, 0, s,
                          "incomplete client cert configured for SSL proxy "

which resulted in a:

  incomplete client cert configured for SSL proxy (missing or encrypted private key?)

error on startup, which was an improvement.

Further investigation using gdb and testing showed that having the private key before the certificate in SSLProxyMachineCertificateFile was triggering the segfault.  Changing this file to have the certificate first resolved the issue.

I'd suggest applying the above patch to avoid the segfault and at least updating the SSLProxyMachineCertificateFile documentation to say that the certificate should come before the private key.

Note that this issue does not occur in Apache/2.2.11 with OpenSSL/0.9.8k but does also occur in Apache/2.2.17 with OpenSSL/1.0.0c.
Comment 1 Orion Poplawski 2012-08-16 22:28:52 UTC
Seeing this on EL6 with mod_ssl-2.2.15-15.sl6.1.i686
Comment 2 Joe Orton 2012-08-17 12:00:58 UTC
Thanks for the patch, Keith. 

I've committed this with an added check that the keypair actually matches.

r1374214
Comment 3 Kaspar Brand 2013-04-13 11:39:20 UTC
(In reply to comment #0)
> Further investigation using gdb and testing showed that having the private
> key before the certificate in SSLProxyMachineCertificateFile was triggering
> the segfault.  Changing this file to have the certificate first resolved the
> issue.
> 
> I'd suggest applying the above patch to avoid the segfault and at least
> updating the SSLProxyMachineCertificateFile documentation to say that the
> certificate should come before the private key.
> 
> Note that this issue does not occur in Apache/2.2.11 with OpenSSL/0.9.8k but
> does also occur in Apache/2.2.17 with OpenSSL/1.0.0c.

This observation is correct. It's actually caused by a regression in OpenSSL 1.0.0 and later, as reported here, in the meantime:

  http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3028

With OpenSSL up to 0.9.8, the order doesn't matter. In this case, however, the check for encrypted private keys is insufficient, as found when dealing with bug 54698.

An additional fix has been committed to trunk in r1467593. Backport proposals for 2.2.x and 2.4.x submitted with r1467594.
Comment 4 Graham Leggett 2013-04-27 22:20:05 UTC
Backported to v2.4.
Comment 5 Kaspar Brand 2013-05-01 06:19:09 UTC
Commit for 2.4.x: r1476685. To appear in 2.4.5.
Comment 6 Kaspar Brand 2013-11-24 09:43:33 UTC
(In reply to Kaspar Brand from comment #3)
> It's actually caused by a regression in OpenSSL
> 1.0.0 and later, as reported here, in the meantime:
> 
>   http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3028

Will be fixed with the next OpenSSL releases, i.e. 1.0.0l, 1.0.1f and 1.0.2 (committed to the repository in early August).