Bug 54698 - Segmentation Fault with SSLProxyMachineCertificateFile
Summary: Segmentation Fault with SSLProxyMachineCertificateFile
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_ssl (show other bugs)
Version: 2.2.24
Hardware: All All
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-03-14 18:08 UTC by alain
Modified: 2013-09-25 13:14 UTC (History)
0 users



Attachments
2.2.x backport of the fixes from PR 52212 (1.78 KB, patch)
2013-03-21 05:47 UTC, Kaspar Brand
Details | Diff
Apache Debug Log (152.96 KB, text/plain)
2013-03-27 21:20 UTC, alain
Details
2.2.x backport of the fixes from PR 52212, amended (1.81 KB, patch)
2013-04-03 06:53 UTC, Kaspar Brand
Details | Diff
proxy-error.log_wrong-private-key (2.24 KB, application/octet-stream)
2013-04-11 21:03 UTC, alain
Details
proxy-error.log_cert-with-keypass (216.15 KB, application/octet-stream)
2013-04-11 21:06 UTC, alain
Details

Note You need to log in before you can comment on or make changes to this bug.
Description alain 2013-03-14 18:08:46 UTC
Client -> Proxy -> WebServer without client auth = OK
Client -> Proxy -> WebServer with client auth & SSLProxyMachineCertificateFile =KO

Apache compiled fron sources:
./configure --prefix=/usr/local/apache2 --enable-module=most --enable-shared=max --enable-rewrite --enable-unique-id --enable-proxy-http --enable-proxy --enable-proxy-connect --enable-ssl

Server version: Apache/2.2.24 (Unix)
Server built:   Mar 14 2013 17:46:34
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local/apache2"
 -D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

root# ldd /usr/local/apache2/bin/httpd
    linux-gate.so.1 =>  (0xb7771000)
    libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb771f000)
    libcrypto.so.0.9.8 => /usr/lib/i686/cmov/libcrypto.so.0.9.8 (0xb75c7000)
    libm.so.6 => /lib/libm.so.6 (0xb75a0000)
    libaprutil-1.so.0 => /usr/lib/libaprutil-1.so.0 (0xb7580000)
    libdb-4.8.so => /usr/lib/libdb-4.8.so (0xb741a000)
    libapr-1.so.0 => /usr/lib/libapr-1.so.0 (0xb73ec000)
    libpthread.so.0 => /lib/libpthread.so.0 (0xb73d3000)
    libc.so.6 => /lib/libc.so.6 (0xb728e000)
    libdl.so.2 => /lib/libdl.so.2 (0xb7289000)
    libz.so.1 => /usr/lib/libz.so.1 (0xb7275000)
    /lib/ld-linux.so.2 (0xb7772000)
    libuuid.so.1 => /lib/libuuid.so.1 (0xb7271000)
    librt.so.1 => /lib/librt.so.1 (0xb7268000)
    libcrypt.so.1 => /lib/libcrypt.so.1 (0xb7236000)
    libexpat.so.1 => /usr/lib/libexpat.so.1 (0xb720f000)


Error Log
[Thu Mar 14 18:45:22 2013] [info] mod_unique_id: using ip addr 192.168.0.77
[Thu Mar 14 18:45:23 2013] [info] Init: Seeding PRNG with 136 bytes of entropy
[Thu Mar 14 18:45:23 2013] [info] Loading certificate & private key of SSL-aware server
[Thu Mar 14 18:45:23 2013] [debug] ssl_engine_pphrase.c(470): unencrypted RSA private key - pass phrase not required
[Thu Mar 14 18:45:23 2013] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Mar 14 18:45:23 2013] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Mar 14 18:45:23 2013] [info] Init: Initializing (virtual) servers for SSL
[Thu Mar 14 18:45:23 2013] [info] Configuring server for SSL protocol
[Thu Mar 14 18:45:23 2013] [debug] ssl_engine_init.c(471): Creating new SSL context (protocols: SSLv3, TLSv1)
[Thu Mar 14 18:45:23 2013] [debug] ssl_engine_init.c(706): Configuring permitted SSL ciphers [HIGH:MEDIUM:!aNULL:!MD5]
[Thu Mar 14 18:45:23 2013] [debug] ssl_engine_init.c(420): Configuring TLS extension handling
[Thu Mar 14 18:45:23 2013] [debug] ssl_engine_init.c(837): Configuring RSA server certificate
[Thu Mar 14 18:45:23 2013] [warn] RSA server certificate CommonName (CN) `proxy.company.com' does NOT match server name!?
[Thu Mar 14 18:45:23 2013] [debug] ssl_engine_init.c(876): Configuring RSA server private key
[Thu Mar 14 18:45:23 2013] [info] mod_ssl/2.2.24 compiled against Server: Apache/2.2.24, Library: OpenSSL/0.9.8o
[Thu Mar 14 18:45:23 2013] [info] mod_unique_id: using ip addr 192.168.0.77
[Thu Mar 14 18:45:24 2013] [info] Init: Seeding PRNG with 136 bytes of entropy
[Thu Mar 14 18:45:24 2013] [info] Loading certificate & private key of SSL-aware server
[Thu Mar 14 18:45:24 2013] [debug] ssl_engine_pphrase.c(470): unencrypted RSA private key - pass phrase not required
[Thu Mar 14 18:45:24 2013] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Mar 14 18:45:24 2013] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(253): shmcb_init allocated 512000 bytes of shared memory
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(272): for 511952 bytes (512000 including header), recommending 32 subcaches, 133 indexes each
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(306): shmcb_init_memory choices follow
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(308): subcache_num = 32
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(310): subcache_size = 15996
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(312): subcache_data_offset = 2144
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(314): subcache_data_size = 13852
[Thu Mar 14 18:45:24 2013] [debug] ssl_scache_shmcb.c(316): index_num = 133
[Thu Mar 14 18:45:24 2013] [info] Shared memory session cache initialised
[Thu Mar 14 18:45:24 2013] [info] Init: Initializing (virtual) servers for SSL
[Thu Mar 14 18:45:24 2013] [info] Configuring server for SSL protocol
[Thu Mar 14 18:45:24 2013] [debug] ssl_engine_init.c(471): Creating new SSL context (protocols: SSLv3, TLSv1)
[Thu Mar 14 18:45:24 2013] [debug] ssl_engine_init.c(706): Configuring permitted SSL ciphers [HIGH:MEDIUM:!aNULL:!MD5]
[Thu Mar 14 18:45:24 2013] [debug] ssl_engine_init.c(420): Configuring TLS extension handling
[Thu Mar 14 18:45:24 2013] [debug] ssl_engine_init.c(837): Configuring RSA server certificate
[Thu Mar 14 18:45:24 2013] [warn] RSA server certificate CommonName (CN) `proxy.company.com' does NOT match server name!?
[Thu Mar 14 18:45:24 2013] [debug] ssl_engine_init.c(876): Configuring RSA server private key
[Thu Mar 14 18:45:24 2013] [info] mod_ssl/2.2.24 compiled against Server: Apache/2.2.24, Library: OpenSSL/0.9.8o
[Thu Mar 14 18:45:24 2013] [warn] pid file /usr/local/apache2/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 1 in child 5507 for worker proxy:reverse
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 1 in child 5507 for (*)
[Thu Mar 14 18:45:24 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/0.9.8o configured -- resuming normal operations
[Thu Mar 14 18:45:24 2013] [info] Server built: Mar 14 2013 17:46:34
[Thu Mar 14 18:45:24 2013] [debug] prefork.c(1023): AcceptMutex: sysvsem (default: sysvsem)
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 1 in child 5509 for worker proxy:reverse
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 1 in child 5510 for worker proxy:reverse
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 1 in child 5509 for (*)
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 1 in child 5510 for (*)
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 1 in child 5511 for worker proxy:reverse
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 1 in child 5508 for worker proxy:reverse
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 1 in child 5511 for (*)
[Thu Mar 14 18:45:24 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 1 in child 5508 for (*)
[Thu Mar 14 18:45:35 2013] [debug] proxy_util.c(1820): proxy: grabbed scoreboard slot 1 in child 5514 for worker proxy:reverse
[Thu Mar 14 18:45:35 2013] [debug] proxy_util.c(1839): proxy: worker proxy:reverse already initialized
[Thu Mar 14 18:45:35 2013] [debug] proxy_util.c(1936): proxy: initialized single connection worker 1 in child 5514 for (*)
[Thu Mar 14 18:45:41 2013] [notice] child pid 5510 exit signal Segmentation fault (11)

(gdb) backtrace
#0  0xb7ef6ff8 in EVP_PKEY_cmp () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#1  0xb7f21cb6 in X509_check_private_key () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#2  0xb7fcd1ed in ?? () from /usr/lib/i686/cmov/libssl.so.0.9.8
#3  0xb7fa9150 in ssl3_send_client_certificate () from /usr/lib/i686/cmov/libssl.so.0.9.8
#4  0xb7facb37 in ssl3_connect () from /usr/lib/i686/cmov/libssl.so.0.9.8
#5  0xb7fc424a in SSL_connect () from /usr/lib/i686/cmov/libssl.so.0.9.8
#6  0xb7fb5b33 in ssl23_connect () from /usr/lib/i686/cmov/libssl.so.0.9.8
#7  0xb7fc424a in SSL_connect () from /usr/lib/i686/cmov/libssl.so.0.9.8
#8  0x080c8043 in ssl_io_filter_connect ()
#9  0x080c8d35 in ssl_io_filter_output ()
#10 0x08093466 in ap_pass_brigade ()
#11 0x080b402e in pass_brigade ()
#12 0x080b483c in stream_reqbody_cl ()
#13 0x080b5ec5 in ap_proxy_http_request ()
#14 0x080b7b82 in proxy_http_handler ()
#15 0x080a7fd0 in proxy_run_scheme_handler ()
#16 0x080a4d7a in proxy_handler ()
#17 0x08087497 in ap_run_handler ()
#18 0x08087bc2 in ap_invoke_handler ()
#19 0x080dc0d2 in ap_process_request ()
#20 0x080d90e5 in ap_process_http_connection ()
#21 0x0808f477 in ap_run_process_connection ()
#22 0x0808f88b in ap_process_connection ()
#23 0x080fdc32 in child_main ()
#24 0x080fdd33 in make_child ()
#25 0x080fe2ce in ap_mpm_run ()
#26 0x08071239 in main ()

Thanks for help,
Tell me if you want more.
Alain
Comment 1 Kaspar Brand 2013-03-16 07:38:53 UTC
What are the contents of the file referenced by SSLProxyMachineCertificateFile? Does the private key appear before the certificate? If so, it might be the same issue as reported in bug 52212 (which hasn't been backported to 2.2.x yet, but as a workaround, you can swap the order of the private key and the cert in the SSLProxyMachineCertificateFile).
Comment 2 alain 2013-03-17 06:09:10 UTC
Thank you for the answer, this issue was solved by users forum, this is because of passphrase on private key.

On documentation, this is written:
Currently there is no support for encrypted private keys

Was not clear for me, may be you could add "private key with passphrase are not supported" as it is usually described.

Thanks, you can close.
Alain
Comment 3 Kaspar Brand 2013-03-17 07:55:30 UTC
(In reply to comment #2)
> Thank you for the answer, this issue was solved by users forum, this is
> because of passphrase on private key.

Are you sure that this was the only/real cause of the problem? There's actually code which should detect encrypted private keys (see bug 24030 and http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/ssl/ssl_engine_init.c?r1=101154&r2=101878).

In your particular case, was the private key encrypted *and* appearing before the cert? (That would make it a duplicate of bug 52212, then.)
Comment 4 alain 2013-03-17 08:36:14 UTC
I don't see any duplicate in 24030 or 52212, my environment was:
- certfile with certificate on top of file and private key at the bottom
- private key with passphrase

In your previous message, I understood normally such config does not provide segfault, right?
Comment 5 Kaspar Brand 2013-03-17 09:12:28 UTC
(In reply to comment #4)
> In your previous message, I understood normally such config does not provide
> segfault, right?

It should not, yes - the purpose of the fix in r101878 (applied in 2003, i.e. well before 2.2.0 was released in December 2005) was to avoid segfaults in this case... but perhaps it was incomplete (or has become incomplete, due to other changes in mod_ssl over the years).

Can you provide the exact PEM headers/footers (BEGIN/END liens) in the SSLProxyMachineCertificateFile you're using (simply leave out the Base64 encoded stuff).
Comment 6 alain 2013-03-17 15:33:48 UTC
I'm not sure I did get you correctly.

Below are headers in the file:
-----BEGIN CERTIFICATE-----
.............
-----END CERTIFICATE-----

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,960B7FA319D6C029
............
-----END RSA PRIVATE KEY-----

Below is the certificate:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 118 (0x76)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=FR, ST=Is\xE8re, L=Grenoble, O=companyname, CN=servername.companyname.com/emailAddress=admin@companyname.com
        Validity
            Not Before: Mar 15 16:13:37 2013 GMT
            Not After : Mar 13 16:13:37 2023 GMT
        Subject: C=FR, ST=Is\xE8re, L=Grenoble, O=companyname, CN=proxy.companyname.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    ..............................
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Client, S/MIME, Object Signing
            Netscape Comment: 
                TinyCA Generated Certificate
            X509v3 Subject Key Identifier: 
                .........................................
            X509v3 Authority Key Identifier: 
                .........................................
                DirName:/C=FR/ST=Is\xE8re/L=Grenoble/O=companyname/CN=servername.companyname.com/emailAddress=admin@companyname.com
                serial:00

            X509v3 Issuer Alternative Name: 
                <EMPTY>

            X509v3 Subject Alternative Name: 
                email:webmaster.companyname.com
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
    Signature Algorithm: sha1WithRSAEncryption
        .............................................
Comment 7 Kaspar Brand 2013-03-20 06:37:46 UTC
(In reply to comment #6)
> I'm not sure I did get you correctly.
> 
> Below are headers in the file:

Thanks, this looks fine. I think it's indeed a problem of the fix from 2003 not being complete enough.

Could you try the following patch with 2.2.24?

http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ssl/ssl_engine_init.c?r1=1358168&r2=1375445&view=patch

With this additional patch, an encrypted private key should no longer cause segfaults when making an SSL proxy connection with client auth, but instead fail at startup with an "incomplete client cert configured for SSL proxy (missing or encrypted private key?)" message.

Based on whether your tests are successful, I would then propose this patch for backporting to 2.2.x (and 2.4.x).
Comment 8 alain 2013-03-20 18:51:29 UTC
Compilation failed:
...................
/usr/share/apr-1.0/build/libtool --silent --mode=compile i486-linux-gnu-gcc -pthread    -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE    -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/srclib/pcre -I. -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/os/unix -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/server/mpm/prefork -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/http -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/filters -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/proxy -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/include -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/generators -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/mappers -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/database -I/usr/include/apr-1.0 -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/proxy/../generators -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/ssl -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/dav/main  -prefer-non-pic -static -c ssl_engine_dh.c && touch ssl_engine_dh.lo
/usr/share/apr-1.0/build/libtool --silent --mode=compile i486-linux-gnu-gcc -pthread    -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE    -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/srclib/pcre -I. -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/os/unix -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/server/mpm/prefork -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/http -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/filters -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/proxy -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/include -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/generators -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/mappers -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/database -I/usr/include/apr-1.0 -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/proxy/../generators -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/ssl -I/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/dav/main  -prefer-non-pic -static -c ssl_engine_init.c && touch ssl_engine_init.lo
ssl_engine_init.c: In function ‘ssl_init_proxy_certs’:
ssl_engine_init.c:1064: error: ‘SSLLOG_MARK’ undeclared (first use in this function)
ssl_engine_init.c:1064: error: (Each undeclared identifier is reported only once
ssl_engine_init.c:1064: error: for each function it appears in.)
ssl_engine_init.c:1065: error: expected ‘)’ before string constant
ssl_engine_init.c:1067: warning: passing argument 3 of ‘ssl_log_ssl_error’ makes integer from pointer without a cast
ssl_private.h:728: note: expected ‘int’ but argument is of type ‘struct server_rec *’
ssl_engine_init.c:1067: error: too few arguments to function ‘ssl_log_ssl_error’
ssl_engine_init.c:1068: error: too many arguments to function ‘ssl_die’
make[3]: *** [ssl_engine_init.lo] Error 1
make[3]: Leaving directory `/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/ssl'
make[2]: *** [install-recursive] Error 1
make[2]: Leaving directory `/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules/ssl'
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory `/root/httpd-2.2.24-PATCH/httpd-2.2.24/modules'
make: *** [install-recursive] Error 1

I verified the patch, looks good:
diff -u /root/httpd-2.2.24/modules/ssl/ssl_engine_init.c ssl_engine_init.c
--- /root/httpd-2.2.24/modules/ssl/ssl_engine_init.c	2012-10-07 08:39:16.000000000 +0200
+++ ssl_engine_init.c	2013-03-20 19:39:48.000000000 +0100
@@ -1051,7 +1051,7 @@
     for (n = 0; n < ncerts; n++) {
         X509_INFO *inf = sk_X509_INFO_value(sk, n);
 
-        if (!inf->x509 || !inf->x_pkey) {
+        if (!inf->x509 || !inf->x_pkey || !inf->x_pkey->dec_pkey) {
             sk_X509_INFO_free(sk);
             ap_log_error(APLOG_MARK, APLOG_STARTUP, 0, s,
                          "incomplete client cert configured for SSL proxy "
@@ -1059,6 +1059,15 @@
             ssl_die();
             return;
         }
+        
+        if (X509_check_private_key(inf->x509, inf->x_pkey->dec_pkey) != 1) {
+            ssl_log_xerror(SSLLOG_MARK, APLOG_STARTUP, 0, ptemp, s, inf->x509,
+                           APLOGNO(02326) "proxy client certificate and "
+                           "private key do not match");
+            ssl_log_ssl_error(SSLLOG_MARK, APLOG_ERR, s);
+            ssl_die(s);
+            return;
+        }
     }
 
     ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
@@ -1070,7 +1079,11 @@
         return;
     }
 
-    /* Load all of the CA certs and construct a chain */
+    /* If SSLProxyMachineCertificateChainFile is configured, load all
+     * the CA certs and have OpenSSL attempt to construct a full chain
+     * from each configured end-entity cert up to a root.  This will
+     * allow selection of the correct cert given a list of root CA
+     * names in the certificate request from the server.  */
     pkp->ca_certs = (STACK_OF(X509) **) apr_pcalloc(p, ncerts * sizeof(sk));
     sctx = X509_STORE_CTX_new();
Comment 9 Kaspar Brand 2013-03-21 05:47:16 UTC
Created attachment 30091 [details]
2.2.x backport of the fixes from PR 52212

(In reply to comment #8)
> I verified the patch, looks good:

My bad, sorry. 2.2 doesn't have ssl_log_xerror etc. Can you try the patch I'm just attaching now instead?

It would also be interesting to test what happens when you put a deliberately "wrong" private key into the file (i.e., one which doesn't match the cert's public key).
Comment 10 alain 2013-03-24 05:42:09 UTC
looks better. Thanks.

1st case: Cert and key do not match

[Sat Mar 23 08:03:38 2013] [debug] ssl_engine_init.c(876): Configuring RSA server private key
[Sat Mar 23 08:03:38 2013] [debug] ssl_engine_init.c(471): Creating new SSL context (protocols: SSLv2, SSLv3, TLSv1)
proxy client certificate and private key do not match
[Sat Mar 23 08:03:38 2013] [error] SSL Library Error: 185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch

2nd case: Cert and key with pass

[Sat Mar 23 08:27:37 2013] [info] [client 192.168.0.53] SSL Proxy connect failed
[Sat Mar 23 08:27:37 2013] [info] SSL Library Error: 336151571 error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported certificate
[Sat Mar 23 08:27:37 2013] [info] [client 192.168.0.53] Connection closed to child 0 with abortive shutdown (server servername.companyname.com:443)
[Sat Mar 23 08:27:37 2013] [info] [client 10.8.0.10] Connection closed to child 1 with standard shutdown (server servername.companyname.com:443)
Comment 11 Kaspar Brand 2013-03-24 12:55:16 UTC
(In reply to comment #10)

Thanks for your tests.

> 2nd case: Cert and key with pass
> 
> [Sat Mar 23 08:27:37 2013] [info] [client 192.168.0.53] SSL Proxy connect
> failed
> [Sat Mar 23 08:27:37 2013] [info] SSL Library Error: 336151571
> error:14094413:SSL routines:SSL3_READ_BYTES:sslv3 alert unsupported
> certificate
> [Sat Mar 23 08:27:37 2013] [info] [client 192.168.0.53] Connection closed to
> child 0 with abortive shutdown (server servername.companyname.com:443)
> [Sat Mar 23 08:27:37 2013] [info] [client 10.8.0.10] Connection closed to
> child 1 with standard shutdown (server servername.companyname.com:443)

This is a bit puzzling - I was expecting a somewhat different behavior... is there no log message saying "incomplete client cert configured for SSL proxy (missing or encrypted private key?)" when you are starting/restarting httpd with a passphrase-protected private key?
Comment 12 alain 2013-03-27 21:20:47 UTC
Created attachment 30108 [details]
Apache Debug Log

 when private key is pass protected and with patch applied
Comment 13 alain 2013-03-27 21:24:34 UTC
Sorry my comment disappeared from previous message, here it is:

This is the Debug log Apache on the proxy.

proxy.companyname.com = 192.168.0.77 = proxy apache
servername.companyname.com = 192.168.0.53 = web server
Comment 14 Kaspar Brand 2013-04-03 06:53:38 UTC
Created attachment 30138 [details]
2.2.x backport of the fixes from PR 52212, amended

(In reply to comment #12)
> Created attachment 30108 [details]
> Apache Debug Log
> 
>  when private key is pass protected and with patch applied

Thanks, meanwhile I found that the check(s) for an encrypted private key are not sufficient for OpenSSL 0.9.8 - I'm attaching an updated patch. Would you mind testing the following combinations with this version of the patch?

a) with encrypted (i.e. passphrase-protected) private key

b) with non-matching private key

c) with unencrypted, matching private key

With a) and b), httpd should now fail to start/restart.
Comment 15 alain 2013-04-11 21:03:51 UTC
Created attachment 30180 [details]
proxy-error.log_wrong-private-key

PR54698_2.2.x_v2.patch
-> apache proxy log when cert with wrong private key and
Comment 16 alain 2013-04-11 21:06:05 UTC
Created attachment 30181 [details]
proxy-error.log_cert-with-keypass

PR54698_2.2.x_v2.patch
-> apache proxy log when cert with key pass
Comment 17 alain 2013-04-11 21:11:01 UTC
Thanks for the patch,
below are my tests:

a) with encrypted (i.e. passphrase-protected) private key
-> see attached log  proxy-error.log_cert-with-keypass
-> I don't agree, apache didn't fail to start ... or did I mismix the keys ???

b) with non-matching private key
-> see attached log   proxy-error.log_wrong-private-key
-> I agree, apache failed to start

c) with unencrypted, matching private key
-> OK, no log provided
Comment 18 Kaspar Brand 2013-04-12 05:06:09 UTC
(In reply to comment #17)
> below are my tests:
> 
> a) with encrypted (i.e. passphrase-protected) private key
> -> see attached log  proxy-error.log_cert-with-keypass
> -> I don't agree, apache didn't fail to start ... or did I mismix the keys
> ???

This is strange... you are using an encrypted key in the format shown in comment 6, is that correct?

Are you sure that patch v2 applied cleanly? Specifically, in ssl_engine_init.c, do lines 1054 ff. look like this?

        if (!inf->x509 || !inf->x_pkey || !inf->x_pkey->dec_pkey ||
            inf->enc_data) {
            sk_X509_INFO_free(sk);
            ap_log_error(APLOG_MARK, APLOG_STARTUP, 0, s,
                         "incomplete client cert configured for SSL proxy "
                         "(missing or encrypted private key?)");

(note the line with "inf->enc_data")
Comment 19 alain 2013-04-12 06:33:00 UTC
Yes, I am using key like in comment 6
Yes, I verified, patch is applied correctly

I will recheck, maybe I mixed the keys. I will regenerate keys and retest. Waiting for non working hours to do the test.
Comment 20 alain 2013-04-13 05:30:33 UTC
Sorry, you were right, apache server does not start in the case of pass phrase protection, below is the error_log of the proxy. I guess I mixed the keys in the previous test.

[Sat Apr 13 07:24:04 2013] [info] mod_unique_id: using ip addr 192.168.0.77
[Sat Apr 13 07:24:05 2013] [info] Init: Seeding PRNG with 136 bytes of entropy
[Sat Apr 13 07:24:05 2013] [info] Loading certificate & private key of SSL-aware server
[Sat Apr 13 07:24:05 2013] [debug] ssl_engine_pphrase.c(470): unencrypted RSA private key - pass phrase not required
[Sat Apr 13 07:24:05 2013] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Sat Apr 13 07:24:05 2013] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Sat Apr 13 07:24:05 2013] [info] Init: Initializing (virtual) servers for SSL
[Sat Apr 13 07:24:05 2013] [debug] ssl_engine_init.c(471): Creating new SSL context (protocols: SSLv2, SSLv3, TLSv1)
incomplete client cert configured for SSL proxy (missing or encrypted private key?)
Comment 21 Kaspar Brand 2013-04-13 11:30:45 UTC
Thanks for verifying.

Fix committed to trunk with r1467593.

Backports for 2.2.x and 2.4.x (including fixes for PR 52212, see comment 1) proposed with r1467594.
Comment 22 Kaspar Brand 2013-05-01 06:20:46 UTC
Commit for 2.4.x: r1476685. To appear in 2.4.5.
Comment 23 Kaspar Brand 2013-09-25 13:14:06 UTC
Fixed in 2.4.6 and 2.2.25, respectively.