The ssl_init_ctx_verify function contains the following if statement which will never evaluate to true according to the current code: if (!ca_list) { * on line 560 of ssl_engine_init.c in version 2.2.3. * on line 547 of ssl_engine_init.c in version 2.0.59. ca_list is initialized with a call to sk_X509_NAME_new in the ssl_init_FindCAList function which is called immediately preceding that if statement in ssl_init_ctx_verify, and there is no possibility of it being set to a NULL or zero value before ssl_init_FindCAList returns it... The end result is that the error message "Unable to determine list of acceptable CA certificates for client authentication" will never make it into an error log, and Apache will start up acting as if the CA certificates were loaded correctly. If "SSLVerifyClient required" is set for the whole virtual host, then users may find this message in their logs: "Init: Oops, you want to request client authentication, but no CAs are known for verification!? [Hint: SSLCACertificate*]" That was particularly confusing to me because Apache never complained about my SSLCACertificateFile and/or SSLCACertificatePath settings. You can verify that the "Unable to determine list of acceptable CA certificates for client authentication" has never really made into users' error logs by googling for that exact phrase. It only yields two or three results. There is also a problem in ssl_init_FindCAList where openssl errors are silently ignored if loading SSLCACertificateFile fails. The ssl_init_PushCAList function is called to add the ca_file's certificate(s) to ca_list, but if its call to SSL_load_client_CA_file comes up empty, then it returns immediately without having added anything to ca_list. That is how it has to be in ssl_init_PushCAList, because it is also called for each file in the SSLCACertificatePath directory if that is also set, and that directory will contain non-certificate files, so printing out the openssl errors inside ssl_init_PushCAList is a bad idea. However, I can find no reason why ssl_init_FindCAList shouldn't check to see if ca_list is still empty after trying to use ssl_init_PushCAList to add the SSLCACertificateFile's certs to the list. If ca_list is still empty after trying to load the cert(s) in that file, then we know openssl has an error to report, and currently that error is being silently discarded. In my case, I had a self-signed CA certificate with "BEGIN TRUSTED CERTIFICATE" on the first line, instead of "BEGIN CERTIFICATE" and openssl was unable to load the certificate because: 6751:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:642:Expecting: CERTIFICATE But I never saw any mention of that in my Apache error logs, which resulted in the hours of confusion and debugging which led me here. I have two very small patch files, one for 2.0-HEAD and one for 2.2-HEAD. I'll append those patches as attachments.
Created attachment 18751 [details] Patch for 2.0.59 which fixes the invalid if condition, and logs openssl errors if SSLCACertificateFile fails to load
Created attachment 18752 [details] Patch for 2.2.3 which fixes the invalid if condition, and logs openssl errors if SSLCACertificateFile fails to load
fixed in trunk in r1026906
1) This seems to be the same issue raised in bug #47329. I, too, lost hours debugging the same problem with a TRUSTED certificate. 2) Bug #47329 contained a suggested patch (by jorton, modified by me) that was never committed. 3) The code added here only reports an error in one of two cases. Your analysis indicated that the directory case should continue to fail silently because other files can exist in that directory. I don't think this is right. If a directory contains hundreds of files, debugging a problem where just one or two fail to load will be at least as painful as the case that you fixed. We can do better. I believe that the only other files that can legitimately be in a certificate directory are CRLs and hash symlinks. Note that c_rehash will complain if any other type of file is found. Hash symlinks for certificates are exactly 8 hex digits, '.', and a decimal sequence number. Filenames of this form can be ignored since the file that they point to will be in the same directory. (At least if the standard c_rehash utility is used. I suppose someone could be 'clever' and separate them.) Ignoring them has the advantage that we will only read the file once, rather than processing it twice and discarding the duplicates... Hash symlinks for CRLs are exactly 8 hex digits, '.r', and a decimal sequence number. Filenames of this form can be ignored since we aren't loading CRLs. A file can contain any number of certificates and/or CRLs. I don't think there's a way to identify what's in a file without reading it and looking for the '^-----BEGIN (((TRUSTED|X509) CERTIFICATE)|X509 CRL)----- line(s). A file that doesn't contain at least one of these tags doesn't belong in the directory, and we should log an error. (It's probably corrupt, in DER format, a key or some other data. Again, c_rehash would whine in this case.) A file that contains just 'X509 CRL' lines can be silently ignored. Otherwise, the file claims to contain one or more certificates, and we can ask OpenSSL to read it. Any error returned by OpenSSL should be logged... It's too bad that OpenSSL doesn't provide a sufficiently detailed error code to avoid the need for mod_ssl to prescan the file, but that's life. This wouldn't be as inefficient as it seems; currently we're asking OpenSSL to process each file twice (once via symlink, once as a regular file). And these are likely to be a good distance apart because of the names. Scanning to classify first is only a string search on each line (not base64/ASN.1 decode, etc), and OpenSSL will almost certainly find the data in the buffer cache for the files we select.
This has been fixed in the 2.4.x branch (in the 2.3.9 dev release)
Closing. 2.2.x branch is EOL and this is fixed in 2.4.x