| Summary: | CPU at 100% in process after SSL "scan" that logs as AH02042 | ||
|---|---|---|---|
| Product: | Apache httpd-2 | Reporter: | mike bayer <mike_mp> |
| Component: | mod_ssl | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | jacob, martyn.shakespeare, sarayut0636064072 |
| Priority: | P2 | ||
| Version: | 2.4.37 | ||
| Target Milestone: | --- | ||
| Hardware: | Macintosh | ||
| OS: | |||
| URL: | admin@apache.org | ||
| Attachments: | cpu graph showing occurrences | ||
|
Description
mike bayer
2019-01-01 23:50:18 UTC
OK yup a kind soul on twitter pointed me to the source of these requests and it is https://www.ssllabs.com/ssltest/analyze.html, I hit my server with this and it's that same IP number 64.41.200.103 and it reliably reproduces the process hanging at 100% CPU when the series of tests gets to about 92%. let me know if you need more information. Nice analysis, thanks. What OpenSSL version? Also can you work out what thread 60 is doing, is it spinning inside OPENSSL_init_crypto() ? this is fedora 29 so packages look like: openssl-1.1.1-3.fc29.x86_64 openssl-pkcs11-0.4.8-2.fc29.x86_64 compat-openssl10-1.0.2o-3.fc29.x86_64 openssl-libs-1.1.1-3.fc29.x86_64 httpd-tools-2.4.37-5.fc29.x86_64 httpd-2.4.37-5.fc29.x86_64 httpd-filesystem-2.4.37-5.fc29.noarch as for what thread 60 is doing, I'm not versed at the moment in stepping through C code with gdb, I would instead hope that this issue is easily reproducible by developers? E.g. create any SSL setup with the above libraries and event MPM (which I have a feeling is not even necessary) and then hit your server with https://www.ssllabs.com/ssltest/analyze.html. Works every time here and per the linked discussion other people are seeing it as well. From my end this kind of looks like a pretty big DOS vulnerability, anyone can just run the attacks from that publicly available online tool a few dozen times against any site running the latest Apache and bring it down. (In reply to mike bayer from comment #3) > this is fedora 29 so packages look like: > > openssl-1.1.1-3.fc29.x86_64 > openssl-pkcs11-0.4.8-2.fc29.x86_64 > compat-openssl10-1.0.2o-3.fc29.x86_64 > openssl-libs-1.1.1-3.fc29.x86_64 > httpd-tools-2.4.37-5.fc29.x86_64 > httpd-2.4.37-5.fc29.x86_64 > httpd-filesystem-2.4.37-5.fc29.noarch Might be related to openssl-1.1.1. I checked a self build 2.4.37 build against RedHat 7's openssl-1.0.2k and there is no spinning. Could this be related to TLS 1.3? Or to the API changes in 1.1.1 that have special handling in the code? This sounds familiar, there are discussions of mod_ssl/openssl 1.1.1 compatibility on the mailing list, related specifically to callback handling. Since those were solved in 2.4.37, contemporaneous to openssl release 1.1.1a (which did not ship with FC29), we may be of limited help. I have not observed the behavior you observe with that specific combination, and have returned to qualsys scanner on many occasions. It may be worth raising a fedora bug on this, and point back to this ticket, since both your httpd 2.4 and openssl 1.1.1 packages are forked. It might also be specific to your (default Fedora?) configuration, would you mind sharing that here (or on a corresponding fedora ticket?) fedora issue with the conf for this vhost is opened at https://bugzilla.redhat.com/show_bug.cgi?id=1664414 This may be based on a misunderstanding by our developers of the SSL_clear_error() function, as first identified here; https://bz.apache.org/bugzilla/show_bug.cgi?id=62590 so... what's the timeline for this to be released and getting it downstream at least as a downloadable rpm? I'm being hit with this issue daily. also any thoughts on why this issue is not more widespread? Proposed in httpd-2.4/STATUS for backport. Committed to branches/2.4.x/ in r1851471 for inclusion in the next 2.4.38 release candidate. *** Bug 63083 has been marked as a duplicate of this bug. *** |