Set up is an Apache 2.2.13 running SSL with client certificate authentication. When running without a defined CRL, it works fine. The client is asked for his certificate in the browser and validation is working. When configuring a CRL for the domain, however, the client is asked for his certificate, and after a few seconds the connection is closed. According to the log file, the child process handling the connection core dumps. The contents of the CRL-file (generated by an external system) is validated with an openssl dump, and it looks fine. The file has been reencoded from DER to PEM using openssl, as the external system generates DER, only.
Please provide a stacktrace of your core dump. See http://httpd.apache.org/dev/debugging.html.
Created attachment 24238 [details] Stack trace of the core file Please note that I have been permitted to supply the certificates and the rest of the configuration files from my security departement, as this is a test environment. Please let me know if this is necessary.
This issue was also present in version 2.2.11.
Pasting the relevant bit of Lars' backtrace #0 0xfe74c124 in ASN1_UTCTIME_print () from /usr/local/ssl/lib/libcrypto.so.0.9.8 (gdb) where #0 0xfe74c124 in ASN1_UTCTIME_print () from /usr/local/ssl/lib/libcrypto.so.0.9.8 #1 0xfe861c60 in ssl_callback_SSLVerify_CRL (ok=1, ctx=0xffbff018, c=0x1a8ae0) at ssl_engine_kernel.c:1498 #2 0xfe861fbc in ssl_callback_SSLVerify (ok=1, ctx=0xffbff018) at ssl_engine_kernel.c:1361 #3 0xfe76386c in internal_verify () from /usr/local/ssl/lib/libcrypto.so.0.9.8 #4 0xfe7641e4 in X509_verify_cert () from /usr/local/ssl/lib/libcrypto.so.0.9.8
When you have opened the core dump with gdb please issue the following commands: frame 1 print bio print crl print crl->crl print crl->crl->lastUpdate print crl->crl->nextUpdate print *(crl->crl->lastUpdate) print *(crl->crl->nextUpdate) Furthermore the crash will go away once you leave the debug log level.
Created attachment 24239 [details] The output of the requested commands.
Not quite sure if I understood you correctly. If I disable debug log level, which is defined inside the ssl virtual host section, it defaults back to the warn level, defined globally. But the core dump still occurs.
(In reply to comment #7) > Not quite sure if I understood you correctly. If I disable debug log level, > which is defined inside the ssl virtual host section, it defaults back to the > warn level, defined globally. But the core dump still occurs. Hm. I guess the crash happens now somewhat later in the code. Can you please crosscheck the stacktrace of the dumps that get created with debug turned off? Nevertheless the following patch should fix the segfault you reported and give better output. Index: modules/ssl/ssl_engine_kernel.c =================================================================== --- modules/ssl/ssl_engine_kernel.c (Revision 813083) +++ modules/ssl/ssl_engine_kernel.c (Arbeitskopie) @@ -1457,10 +1457,20 @@ X509_NAME_print(bio, issuer, 0); BIO_printf(bio, ", lastUpdate: "); - ASN1_UTCTIME_print(bio, X509_CRL_get_lastUpdate(crl)); + if (X509_CRL_get_lastUpdate(crl)) { + ASN1_UTCTIME_print(bio, X509_CRL_get_lastUpdate(crl)); + } + else { + BIO_printf(bio, "NULL"); + } BIO_printf(bio, ", nextUpdate: "); - ASN1_UTCTIME_print(bio, X509_CRL_get_nextUpdate(crl)); + if (X509_CRL_get_nextUpdate(crl)) { + ASN1_UTCTIME_print(bio, X509_CRL_get_nextUpdate(crl)); + } + else { + BIO_printf(bio, "NULL"); + } n = BIO_read(bio, buff, sizeof(buff) - 1); buff[n] = '\0'; @@ -1492,9 +1502,9 @@ /* * Check date of CRL to make sure it's not expired */ - i = X509_cmp_current_time(X509_CRL_get_nextUpdate(crl)); - - if (i == 0) { + if ((X509_CRL_get_nextUpdate(crl) == NULL) + || ((i = X509_cmp_current_time(X509_CRL_get_nextUpdate(crl))) + == 0)) { ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s, "Found CRL has invalid nextUpdate field"); But your debug outputs show that the nextUpdate field of your CRL is empty which is IMHO bad. So I guess your CRL needs fixing as well.
> But your debug outputs show that the nextUpdate field of your CRL is empty > which is IMHO bad. So I guess your CRL needs fixing as well. This is permitted by RFC3280 and openssl can generate the CRL without this field. TBSCertList ::= SEQUENCE { version Version OPTIONAL, -- if present, MUST be v2 signature AlgorithmIdentifier, issuer Name, thisUpdate Time, nextUpdate Time OPTIONAL, revokedCertificates SEQUENCE OF SEQUENCE {
(In reply to comment #9) > > But your debug outputs show that the nextUpdate field of your CRL is empty > > which is IMHO bad. So I guess your CRL needs fixing as well. > > This is permitted by RFC3280 and openssl can generate the CRL without this > field. > > TBSCertList ::= SEQUENCE { > version Version OPTIONAL, > -- if present, MUST be v2 > signature AlgorithmIdentifier, > issuer Name, > thisUpdate Time, > nextUpdate Time OPTIONAL, > revokedCertificates SEQUENCE OF SEQUENCE { Thanks for the info, but how should mod_ssl behave in this case? My patch would cause it to error out. Should we treat the CRL as expired or valid or what?
(In reply to comment #10) > (In reply to comment #9) > > > But your debug outputs show that the nextUpdate field of your CRL is empty > > > which is IMHO bad. So I guess your CRL needs fixing as well. > > > > This is permitted by RFC3280 and openssl can generate the CRL without this > > field. > > > > TBSCertList ::= SEQUENCE { > > version Version OPTIONAL, > > -- if present, MUST be v2 > > signature AlgorithmIdentifier, > > issuer Name, > > thisUpdate Time, > > nextUpdate Time OPTIONAL, > > revokedCertificates SEQUENCE OF SEQUENCE { > > Thanks for the info, but how should mod_ssl behave in this case? My patch would > cause it to error out. Should we treat the CRL as expired or valid or what? Whoops, it's more complicated, section 5.0: Conforming CAs are not required to issue CRLs if other revocation or certificate status mechanisms are provided. When CRLs are issued, the CRLs MUST be version 2 CRLs, include the date by which the next CRL will be issued in the nextUpdate field (section 5.1.2.5), include the CRL number extension (section 5.2.3), and include the authority key identifier extension (section 5.2.1). later: This profile requires inclusion of nextUpdate in all CRLs issued by conforming CRL issuers ... The behavior of clients processing CRLs which omit nextUpdate is not specified by this profile. Iff there's no "version" extension in the CRL I suspect we should treat nextUpdate == NULL as valid, but version >1 and nextUpdate == NULL looks like it should be configurable
> Thanks for the info, but how should mod_ssl behave in this case? My patch would > cause it to error out. Should we treat the CRL as expired or valid or what? Please note that this CRL is read by another system today - a Cisco CSS. I'm working on moving this part of the functionality to Apache. I will not interfere with your discussion on the RFC as it is not my field, but I'm pretty sure that the CRL is being treated as valid - an "open" expiry date, so to speak. Regarding the stack trace of the cores with the debug parameter off, I will perform them tomorrow, as I suddenly have no access to the system core dump directory. Can't seem to override where to dump the core in the Apache config file (parameter has no effect).
Created attachment 24242 [details] The contents of the CRL. This file shows the contents of the CRL. Don't know if it shows anything you haven't already figured out.
I finally got to run the stack trace on a warn log level. Seems the same (with a slight diff in line number on the last one): #0 0xfe762fc4 in X509_cmp_time () from /usr/local/ssl/lib/libcrypto.so.0.9.8 #1 0xfe861b80 in ssl_callback_SSLVerify_CRL (ok=1, ctx=0xffbfef48, c=0x19ae50) at ssl_engine_kernel.c:1530 #2 0xfe861fbc in ssl_callback_SSLVerify (ok=1, ctx=0xffbfef48) at ssl_engine_kernel.c:1361 #3 0xfe76386c in internal_verify () from /usr/local/ssl/lib/libcrypto.so.0.9.8 #4 0xfe7641e4 in X509_verify_cert () from /usr/local/ssl/lib/libcrypto.so.0.9.8
(In reply to comment #14) > I finally got to run the stack trace on a warn log level. > > Seems the same (with a slight diff in line number on the last one): > > #0 0xfe762fc4 in X509_cmp_time () from /usr/local/ssl/lib/libcrypto.so.0.9.8 > #1 0xfe861b80 in ssl_callback_SSLVerify_CRL (ok=1, ctx=0xffbfef48, c=0x19ae50) > at ssl_engine_kernel.c:1530 That is what I expected. Did you test the patch? Admittedly it only removes the segfault but does not work with your CRL. See the comments from Eric regarding the discussion what should be the right thing to do in this case (empty next update field). This still needs to be decided and worked out into code.
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.