Bug 21160 - SSL certificate chain handling suddenly fails to work properly
Summary: SSL certificate chain handling suddenly fails to work properly
Status: CLOSED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_ssl (show other bugs)
Version: 2.0.48
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
: 13585 (view as bug list)
Depends on:
Blocks:
 
Reported: 2003-06-28 03:04 UTC by David Tonhofer
Modified: 2004-11-16 19:05 UTC (History)
3 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Tonhofer 2003-06-28 03:04:01 UTC
There is as yet not much information here, I will have to try a few things
first (next week, not today it's about 05:00). But here is what happens:

Apache has been configured with three IP-based virtual servers on three
different IP addresses. On each of these addresses, we have an SSL server, thus
three SSL servers in total. 

One with a self-signed root CA certificate   ROOT->C1->SSL virtual host
Two with an 'official' CA certificate        ROOT->C1->C2->SSL virtual host

Everything has been configured, Apache has been happily chugging along...

But then...

After a restart, Apache goes through the SSL virtual servers and asks the
password for each of the three private keys (good). After this, it fails (bad)
with the following error in the error log:

"Failed to configure CA certificate chain!"

(Some additional info would have been of use, too)

The weird thing is that the configuration for SSL had not changed at all. Thus
the production server was suddenly dead in the water w/o reason.

Also, each of the SSL virtual servers work if they are the only ones in the
config file. Certain pairs also work, but not all.

Finally, 'openssl verify' does not find anything amiss with the CA chains.

So, that's all for now. More to follow (hopefully)

What is this server:

Apache/2.0.45 + mod_ssl/2.0.45 + OpenSSL/0.9.7b 

on a RH7.3 OS with gcc-2.96-110 and glibc-2.2.5-39
Comment 1 David Tonhofer 2003-06-30 22:17:52 UTC
As promised, more information (I am actually keeping my word for once, wow!):

I finally got it to work, though why it *does* work is a mystery.

First, some info on what does not work:

I tried the three SSL virtual servers pairwise. On each occasion, Apache startup
failed. I got the ide of setting the verbosity level to debug ('LogLevel Debug'),
thus we find the following in the logfile, in case all three SSL virtual servers
are configured:

[Mon Jun 30 23:03:31 2003] [info] Init: Initializing OpenSSL library
[Mon Jun 30 23:03:31 2003] [info] Init: Seeding PRNG with 648 bytes of entropy
[Mon Jun 30 23:03:31 2003] [info] Loading certificate & private key of SSL-aware
server
[Mon Jun 30 23:03:31 2003] [info] Init: Requesting pass phrase via builtin
terminal dialog
[Mon Jun 30 23:03:38 2003] [debug] ssl_engine_pphrase.c(499): encrypted RSA
private key - pass phrase requested
[Mon Jun 30 23:03:38 2003] [info] Loading certificate & private key of SSL-aware
server
[Mon Jun 30 23:03:38 2003] [info] Init: Requesting pass phrase via builtin
terminal dialog
[Mon Jun 30 23:03:47 2003] [debug] ssl_engine_pphrase.c(499): encrypted RSA
private key - pass phrase requested
[Mon Jun 30 23:03:47 2003] [info] Loading certificate & private key of SSL-aware
server
[Mon Jun 30 23:03:47 2003] [info] Init: Requesting pass phrase via builtin
terminal dialog
[Mon Jun 30 23:03:54 2003] [debug] ssl_engine_pphrase.c(499): encrypted RSA
private key - pass phrase requested
[Mon Jun 30 23:03:54 2003] [info] Init: Wiped out the queried pass phrases from
memory
[Mon Jun 30 23:03:54 2003] [info] Init: Generating temporary RSA private keys
(512/1024 bits)
[Mon Jun 30 23:03:54 2003] [info] Init: Generating temporary DH parameters
(512/1024 bits)
[Mon Jun 30 23:03:54 2003] [debug] ssl_scache_dbm.c(422): Inter-Process Session
Cache (DBM) Expiry: old: 0, new: 0, removed: 0
[Mon Jun 30 23:03:54 2003] [info] Init: Initializing (virtual) servers for SSL
[Mon Jun 30 23:03:54 2003] [info] Configuring server for SSL protocol
[Mon Jun 30 23:03:54 2003] [debug] ssl_engine_init.c(436): Creating new SSL
context (protocols: SSLv2, SSLv3, TLSv1)
[Mon Jun 30 23:03:54 2003] [debug] ssl_engine_init.c(611): Configuring permitted
SSL ciphers [ALL:!IDEA:!ADH:EXPORT56:EXPORT40:!NULL:+HIGH:+MEDIUM:+LOW]
[Mon Jun 30 23:03:54 2003] [error] Failed to configure CA certificate chain!

I will spare you the pairs, it's the same...

I then tried each of the SSL virtual servers alone. In each case, 
startup was a success:

[Mon Jun 30 23:03:31 2003] [info] Init: Initializing OpenSSL library
[Mon Jun 30 23:03:31 2003] [info] Init: Seeding PRNG with 648 bytes of entropy
[Mon Jun 30 23:03:31 2003] [info] Loading certificate & private key of SSL-aware
server
[Mon Jun 30 23:03:31 2003] [info] Init: Requesting pass phrase via builtin
terminal dialog
[Mon Jun 30 23:03:38 2003] [debug] ssl_engine_pphrase.c(499): encrypted RSA
private key - pass phrase requested
[Mon Jun 30 23:03:38 2003] [info] Loading certificate & private key of SSL-aware
server
[Mon Jun 30 23:03:38 2003] [info] Init: Requesting pass phrase via builtin
terminal dialog
[Mon Jun 30 23:03:47 2003] [debug] ssl_engine_pphrase.c(499): encrypted RSA
private key - pass phrase requested
[Mon Jun 30 23:03:47 2003] [info] Loading certificate & private key of SSL-aware
server
[Mon Jun 30 23:03:47 2003] [info] Init: Requesting pass phrase via builtin
terminal dialog
[Mon Jun 30 23:03:54 2003] [debug] ssl_engine_pphrase.c(499): encrypted RSA
private key - pass phrase requested
[Mon Jun 30 23:03:54 2003] [info] Init: Wiped out the queried pass phrases from
memory
[Mon Jun 30 23:03:54 2003] [info] Init: Generating temporary RSA private keys
(512/1024 bits)
[Mon Jun 30 23:03:54 2003] [info] Init: Generating temporary DH parameters
(512/1024 bits)
[Mon Jun 30 23:03:54 2003] [debug] ssl_scache_dbm.c(422): Inter-Process Session
Cache (DBM) Expiry: old: 0, new: 0, removed: 0
[Mon Jun 30 23:03:54 2003] [info] Init: Initializing (virtual) servers for SSL
[Mon Jun 30 23:03:54 2003] [info] Configuring server for SSL protocol
[Mon Jun 30 23:03:54 2003] [debug] ssl_engine_init.c(436): Creating new SSL
context (protocols: SSLv2, SSLv3, TLSv1)
[Mon Jun 30 23:03:54 2003] [debug] ssl_engine_init.c(611): Configuring permitted
SSL ciphers [ALL:!IDEA:!ADH:EXPORT56:EXPORT40:!NULL:+HIGH:+MEDIUM:+LOW]
[Mon Jun 30 23:03:54 2003] [error] Failed to configure CA certificate chain!

I figured I would continue with a pair of servers and whittle down the
SSL config file until things began to work. This actually paid off!

It turns that the presence of this block seems to be confusing:

  <Files ~ "\.(cgi|shtml|phtml|php3?)$">
      SSLOptions +StdEnvVars
  </Files>
  <Directory "/usr/local/apache2/cgi-bin">
      SSLOptions +StdEnvVars
  </Directory>

I had this block in each of the three SSL virtual servers, taken from the
original file coming with Apache. I commented it out in one (1) of the three.

Lo and behold! It works! Now the passphrase dialog spits out an error after
having asked for the 2nd passphrase. This however, does not prevent it from
reading the third passpharse. It is also a Good Sign, because whenever this
error shows up, the webserver will be able to configure itself:


Server www.m-plify.com:443 (RSA)
Enter pass phrase:

Server rei1.m-plify.net:443 (RSA)
Enter pass phrase:1024:error:0D094068:asn1 encoding routines:d2i_ASN1_SET:bad
tag:a_set.c:179:
1024:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:939:
1024:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1
error:tasn_dec.c:304:Type=RSA
1024:error:0D09A00D:asn1 encoding routines:d2i_PrivateKey:ASN1 lib:d2i_pr.c:96:


I'm completely at a loss to explain a relationship between the configuration
instructions above and SSL certificate chain configuration, sorry....but that's
what happened.
Comment 2 David Tonhofer 2003-06-30 23:11:09 UTC
Naah, forget what I said about the workaround. The exact same file that
earlier worked now fails to work (yeah, its the fscking SAME file). This must
have to do with the moon phases! OMG...
Comment 3 David Tonhofer 2003-07-01 00:41:31 UTC
I have tried with Apache 2.0.46. There were 3 successfull starts and 1 
unsuccessful one. Not too bad. I swear I will give up all this Computer
Crap and I'm gonna raise sheep in New Zealand!

Anyway, that's it for now. -OO-
Comment 4 kris.verbeeck 2003-07-01 06:53:57 UTC
Same problem here.  A configuration with a certificate chain and two virtual
hosts worked on one system (always) but failed on another (always).  On the
system where it failed, removing one of the virtual hosts fixed the problem.

Setup: Apache 2.0.46 + OpenSSL 0.9.6i
Success system: Gentoo Linux
Failure system: RedHat Advanced Server

I'm not sure whether it is related to the OS vendor.  Will do some more checks
when I get the time.
Comment 5 David Tonhofer 2004-02-09 17:45:07 UTC
I have tried with Apache 2.0.47 and openssl-0.9.7b. Same problem.

And the workaround is (tadaa!):

DO NOT ENCRYPT THE SERVER PRIVATE KEYS.

Arf!
Comment 6 Joe Orton 2004-02-09 22:26:14 UTC
There is a bug which means the OpenSSL error stack is not cleared: I thought
this was a purely cosmetic issue (it causes the error dumps you see during
pphrase entry), but in fact it may well be the cause of this bug:

Can anyone who can reproduce this try the following patch:

http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/ssl/ssl_engine_pphrase.c?r1=1.44&r2=1.45
Comment 7 kris.verbeeck 2004-02-10 10:59:51 UTC
I was able to reproduce the 'Failed to configure CA certificate chain' error
message.  I started Apache and entered a wrong passphrase for the private key
and got "Error: Pass phrase incorrect (5 more retries permitted).".  After that
I entered the correct passphrase and got "Ok: Pass Phrase Dialog successful.". 
Then Apache failed to start (the other virtual hosts probably) and the error log
contained the certificate chain error.  When I enter the correct passphrase from
the beginning everything works allright.

Then I patched the server with the patch given below and retested like above. 
Apache now started succesfully.  So it seems that this error stack clearing
really is more than only cosmetic :)).
Comment 8 Joe Orton 2004-02-10 11:23:17 UTC
Wonderful, thanks Kris.  I've proposed the fix for inclusion in the next 2.0
release.  Thanks for the reports.
Comment 9 Joe Orton 2004-03-10 18:16:23 UTC
*** Bug 13585 has been marked as a duplicate of this bug. ***
Comment 10 Joe Orton 2004-06-10 15:09:01 UTC
*** Bug 29496 has been marked as a duplicate of this bug. ***