Bug 44940

Summary:	Httpd randomly breaks when verifying client certificates
Product:	Apache httpd-2	Reporter:	Chris Cunningham <cunningham.c>
Component:	mod_ssl	Assignee:	Apache HTTPD Bugs Mailing List <bugs>
Status:	RESOLVED LATER
Severity:	major	Keywords:	MassUpdate
Priority:	P2
Version:	2.2.12
Target Milestone:	---
Hardware:	PC
OS:	Linux

Description Chris Cunningham 2008-05-06 02:24:53 UTC

Httpd is configured to accept client certificates, which have been signed by my internal certificate chain:

SSLCACertificateFile  /etc/pki/internal_certificates/caCertificateList.pem
SSLVerifyClient optional
SSLVerifyDepth  3

caCertificateList contains the chain of certificates for the client certificates, with a self-signed root certificate.

This setup works fine in the general case; there is no problem verifying certificates. This suggests that it is not a problem with configuration. The problem occurs after the server has been in use for a few days, having served some tens of thousands of requests - certificate verification suddenly starts failing with:

Certificate Verification: Error (2): unable to get issuer certificate

This begins happening for all client certificates, including those that were previously accepted. Nothing further useful is logged, even with SSL debug. I cannot find a trigger for it; it doesn't seem to be after a particular time form startup or anything like that.

Reloading httpd fixes the problem, for another few days.

Comment 1 Ruediger Pluem 2008-05-06 04:42:45 UTC

Which version of openssl are you using?
Does the same happen with httpd 2.2.8?

Comment 2 Chris Cunningham 2010-08-26 12:00:28 UTC

We're still seeing this problem with Apache 2.2.12, openssl 0.9.8b.

It appears to happen (occasionally) after doing a reload. Doing a restart clears the problem.

Comment 3 Chris Cunningham 2010-08-27 07:44:25 UTC

On further investigation, these appear to happen after a crash in httpd-worker in libperl (it's a mod_perl application). It would seem that after recovering from the crash, apache is in some kind of stuck state where it cannot read the ca certificate?

 I can paste the full log output if it's useful, but it starts off:

*** glibc detected *** /usr/sbin/httpd.worker: double free or corruption (fasttop): 0x00007f90e7133d90 ***
======= Backtrace: =========
/lib64/libc.so.6[0x7f90e20bccec]
/lib64/libc.so.6(cfree+0x8c)[0x7f90e20c090c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_newCONSTSUB+0x14c)[0x7f90dd263ae3]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_anoncode+0x6d)[0x7f90dd2b94f5]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0x141)[0x7f90dd2833cd]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x7dc)[0x7f90dd23d0cb]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_list+0x2e8)[0x7f90dd23d632]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_newATTRSUB+0xf6e)[0x7f90dd26f09c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_utilize+0x24a)[0x7f90dd26d66c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_yyparse+0x1026)[0x7f90dd2609b6]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so[0x7f90dd2d051c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_require+0xb8b)[0x7f90dd2d6c0d]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0x141)[0x7f90dd2833cd]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x7dc)[0x7f90dd23d0cb]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_list+0x2e8)[0x7f90dd23d632]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_newATTRSUB+0xf6e)[0x7f90dd26f09c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_utilize+0x24a)[0x7f90dd26d66c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_yyparse+0x1026)[0x7f90dd2609b6]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so[0x7f90dd2d051c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_require+0xb8b)[0x7f90dd2d6c0d]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0x141)[0x7f90dd2833cd]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x7dc)[0x7f90dd23d0cb]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_list+0x2e8)[0x7f90dd23d632]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_newATTRSUB+0xf6e)[0x7f90dd26f09c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_utilize+0x24a)[0x7f90dd26d66c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_yyparse+0x1026)[0x7f90dd2609b6]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so[0x7f90dd2d051c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_require+0xb8b)[0x7f90dd2d6c0d]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0x141)[0x7f90dd2833cd]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x7dc)[0x7f90dd23d0cb]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_list+0x2e8)[0x7f90dd23d632]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_newATTRSUB+0xf6e)[0x7f90dd26f09c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_utilize+0x24a)[0x7f90dd26d66c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_yyparse+0x1026)[0x7f90dd2609b6]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so[0x7f90dd2d051c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_require+0xb8b)[0x7f90dd2d6c0d]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0x141)[0x7f90dd2833cd]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x7dc)[0x7f90dd23d0cb]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_list+0x2e8)[0x7f90dd23d632]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_newATTRSUB+0xf6e)[0x7f90dd26f09c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_utilize+0x24a)[0x7f90dd26d66c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_yyparse+0x1026)[0x7f90dd2609b6]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so[0x7f90dd2d051c]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_require+0xb8b)[0x7f90dd2d6c0d]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0x141)[0x7f90dd2833cd]
/usr/lib64/perl5/5.8.6/x86_64-linux-thread-multi/CORE/libperl.so(Perl_eval_sv+0x344)[0x7f90dd23ba21]
/etc/httpd/modules/mod_perl.so(modperl_require_module+0x1b0)[0x7f90dd471009]
/etc/httpd/modules/mod_perl.so(modperl_mgv_resolve+0x50b)[0x7f90dd475b41]
/etc/httpd/modules/mod_perl.so(modperl_handler_resolve+0x5a)[0x7f90dd46ed79]
/etc/httpd/modules/mod_perl.so(modperl_callback+0x4e)[0x7f90dd46d6ae]
/etc/httpd/modules/mod_perl.so(modperl_callback_run_handlers+0x2ac)[0x7f90dd46df11]
/etc/httpd/modules/mod_perl.so(modperl_callback_per_dir+0x2f)[0x7f90dd46e42e]
/etc/httpd/modules/mod_perl.so[0x7f90dd468b28]
/etc/httpd/modules/mod_perl.so(modperl_response_handler+0x90)[0x7f90dd468eba]
/usr/sbin/httpd.worker(ap_run_handler+0x7a)[0x7f90e4063a6a]
/usr/sbin/httpd.worker(ap_invoke_handler+0x7c)[0x7f90e4066dcc]
/usr/sbin/httpd.worker(ap_process_request+0x1a8)[0x7f90e4071798]
/usr/sbin/httpd.worker[0x7f90e406e990]
/usr/sbin/httpd.worker(ap_run_process_connection+0x72)[0x7f90e406ad52]
/usr/sbin/httpd.worker[0x7f90e4076395]
/lib64/libpthread.so.0[0x7f90e25ab367]
/lib64/libc.so.6(clone+0x6d)[0x7f90e211e09d]

Comment 4 William A. Rowe Jr. 2018-11-07 21:09:24 UTC

Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.