| Summary: | Certificate list in mod_ssl module context are not sorted, causes segfault | ||
|---|---|---|---|
| Product: | Apache httpd-2 | Reporter: | Pascal Buchbinder <pascal.buchbinder> |
| Component: | mod_ssl | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
| Status: | RESOLVED LATER | ||
| Severity: | normal | Keywords: | MassUpdate |
| Priority: | P2 | ||
| Version: | 2.2.4 | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | other | ||
change subject from "Certificate list in mod_ssl module context are not sorted" to "Certificate list in mod_ssl module context are not sorted, causes segfault" Can you please provide a backtrace of the crash (see http://httpd.apache.org/dev/debugging.html)? ssl_io_filter_input() ssl_io_filter_connect() SSL_accept() ssl23_accept() ssl23_get_client_hello() SSL_accept() ssl3_accept() ssl3_send_server_certificate() ssl3_output_cert_chain() X509_STORE_get_by_subject() X509_OBJECT_retrieve_by_subject() X509_OBJECT_idx_by_subject() sk_find() internal_find() sk_sort() qsort() => x509_object_cmp() It would be helpful to have one with line numbers and the parameters supplied to the functions. Well, it does not matter where the segmentation fault happens. The problem is, that the object (openssl stack objects withn the ssl context) is shared between processes/threads. This object is accessed readonly only, therefore no mutex is required when accessing it. The problem is, that openssl does a sort (changing the order of the objects within the stack for fast access) when accessing the stack the very first time. In order to prevent this sort when accessing the stack from multiple parallel request, I suggest to force a stack sort at server startup (right after all certificates has been loaded), e.g. in the function ssl_init_ctx_verify(). Well, this is a minor change request at all (the worst thing that happens is probably a segfault by some child processes at sever startup). Many thanks... I'm apparently experiencing the same problem:
# apache2ctl restart
/usr/sbin/apache2ctl: line 83: 14407 Segmentation fault $HTTPD ${APACHE_ARGUMENTS} -t 2>/dev/null
/usr/sbin/apache2ctl: line 83: 14388 Segmentation fault $HTTPD ${APACHE_ARGUMENTS} -t
# apache2 -v
Server version: Apache/2.2.8 (Debian)
Server built: Jan 17 2008 21:31:10
Segmentation fault
I'm running gcc 4.1.3, everything's 64bit.
If possible, I will provide whatever information is needed, as directed.
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated. |
mod_ssl re-uses its module context for each request/connection. Example: static void ssl_init_ctx_cipher_suite(server_rec *s, apr_pool_t *p, apr_pool_t *ptemp, modssl_ctx_t *mctx) { SSL_CTX *ctx = mctx->ssl_ctx; This context is accessed as "read only" and can therefore been shared between threads. OpenSSL uses mutexes when accessing global objects (e.g. random generation). The problem I encounter: the server certificates are stored in OpenSSL stacks. The objects in this stack need to be sorted when they get accessed the very first time (sk_find() brings the objects in the right order using qsort()). Stack trace: ssl_io_filter_input() ssl_io_filter_connect() SSL_accept() ssl23_accept() ssl23_get_client_hello() SSL_accept() ssl3_accept() ssl3_send_server_certificate() ssl3_output_cert_chain() X509_STORE_get_by_subject() X509_OBJECT_retrieve_by_subject() X509_OBJECT_idx_by_subject() sk_find() internal_find() sk_sort() qsort() x509_object_cmp() When starting multiple requests (new ssl handshakes) in parallel right after a server restart, the server might crash due multiple threads are accessing the certificate stack which has not been sorted yet (segmentation fault in x509_object_cmp() due the move of the certificate objects in the stack order). Possible workaround: Manual sort of the stacks in the ssl context at server startup, e.g. in mod_ssl ssl_init_ctx_verify() Example: if(ctx->cert_store->objs->comp) { sk_sort(ctx->cert_store->objs); } Impact of this issue is not very high due: - it can only happen after a server restart - may cause a crash of one single server child process - happens only in a multithreaded environment (MPM worker)