Bug 63052

Summary: CPU at 100% in process after SSL "scan" that logs as AH02042
Product: Apache httpd-2 Reporter: mike bayer <mike_mp>
Component: mod_sslAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: normal CC: jacob, martyn.shakespeare, sarayut0636064072
Priority: P2    
Version: 2.4.37   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
URL: admin@apache.org
Attachments: cpu graph showing occurrences

Description mike bayer 2019-01-01 23:50:18 UTC 
Created attachment 36357 [details]
cpu graph showing occurrences

I'm running mod_ssl on a server with about a dozen vhosts and the event MPM.

every few weeks one of the child forks goes to 100% CPU and stops serving requests, and is unresponsive until kill -9'ed.

After a few occurrences, I've isolated a bunch of symptoms.  These symptoms line up exactly with what I've googled at this forum discussion here: https://community.qualys.com/thread/19027-apache-threads-stuck-at-100-after-scan, where they claim it corresponds to something called a "qualys scan".

I don't know what a "Qualys scan" is but as these are public sites I'm running, someone can certainly be doing that to me, as it seems this comes from one set of IP addresses.  

in the logs can be seen:

./alembic.sqlalchemy.org/logs/error_log:[Thu Dec 06 09:51:29.574388 2018] [ssl:error] [pid 32339:tid 140324684879616] [client 64.41.200.104:43236] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Thu Dec 06 09:51:39.791714 2018] [ssl:error] [pid 18719:tid 140324793984768] [client 64.41.200.104:46964] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Mon Dec 17 17:46:26.713047 2018] [ssl:error] [pid 10740:tid 139985525073664] [client 149.202.170.250:40663] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Tue Dec 18 22:56:15.748275 2018] [ssl:error] [pid 10740:tid 139985625786112] [client 64.41.200.104:55826] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Tue Dec 18 22:56:25.981016 2018] [ssl:error] [pid 10741:tid 139985617393408] [client 64.41.200.104:33424] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Wed Dec 19 10:53:22.513141 2018] [ssl:error] [pid 1105:tid 140393639253760] [client 64.41.200.102:41958] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Wed Dec 19 10:53:32.737070 2018] [ssl:error] [pid 727:tid 140393748358912] [client 64.41.200.102:43920] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Tue Jan 01 13:24:32.033175 2019] [ssl:error] [pid 315:tid 140333199312640] [client 64.41.200.103:58752] AH02042: rejecting client initiated renegotiation
./alembic.sqlalchemy.org/logs/error_log:[Tue Jan 01 13:24:42.270493 2019] [ssl:error] [pid 9095:tid 140333437437696] [client 64.41.200.103:33332] AH02042: rejecting client initiated renegotiation


I've done a ptrace on the process in question, and in the particular run, thread 60 seemed to be the one with the problem, so a bunch of snapshots of this thread are below.   additionally attached a graph showing CPU spiking over the month in which these requests occurred.

Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa2023915db in OPENSSL_init_crypto () from target:/lib64/libcrypto.so.1.1
#1  0x00007fa202369506 in ERR_get_state () from target:/lib64/libcrypto.so.1.1
#2  0x00007fa20236970d in ERR_clear_error () from target:/lib64/libcrypto.so.1.1
#3  0x00007fa2025a0403 in ?? () from target:/etc/httpd/modules/mod_ssl.so
#4  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#5  0x000055fb61569f95 in ap_rgetline_core ()
#6  0x000055fb6156a73d in ap_get_mime_headers_core ()
#7  0x000055fb6156d5a4 in ap_read_request ()
#8  0x000055fb61597355 in ?? ()
#9  0x000055fb6158d7f8 in ap_run_process_connection ()
#10 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#11 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#12 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#13 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6

Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202391535 in OPENSSL_init_crypto () from target:/lib64/libcrypto.so.1.1
#1  0x00007fa202369506 in ERR_get_state () from target:/lib64/libcrypto.so.1.1
#2  0x00007fa20236970d in ERR_clear_error () from target:/lib64/libcrypto.so.1.1
#3  0x00007fa2025a0403 in ?? () from target:/etc/httpd/modules/mod_ssl.so
#4  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#5  0x000055fb61569f95 in ap_rgetline_core ()
#6  0x000055fb6156a73d in ap_get_mime_headers_core ()
#7  0x000055fb6156d5a4 in ap_read_request ()
#8  0x000055fb61597355 in ?? ()
#9  0x000055fb6158d7f8 in ap_run_process_connection ()
#10 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#11 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#12 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#13 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6

Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa203001a93 in pthread_getspecific () from target:/lib64/libpthread.so.0
#1  0x00007fa202369543 in ERR_get_state () from target:/lib64/libcrypto.so.1.1
#2  0x00007fa2023697d5 in ?? () from target:/lib64/libcrypto.so.1.1
#3  0x00007fa202525f3e in SSL_get_error () from target:/lib64/libssl.so.1.1
#4  0x00007fa2025a045a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#5  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#6  0x000055fb61569f95 in ap_rgetline_core ()
#7  0x000055fb6156a73d in ap_get_mime_headers_core ()
#8  0x000055fb6156d5a4 in ap_read_request ()
#9  0x000055fb61597355 in ?? ()
#10 0x000055fb6158d7f8 in ap_run_process_connection ()
#11 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#12 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#13 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#14 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6

Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202eab76a in _int_malloc () from target:/lib64/libc.so.6
#1  0x00007fa202eacc8a in malloc () from target:/lib64/libc.so.6
#2  0x00007fa20251170c in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202510f36 in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#6  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#7  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#8  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#9  0x000055fb61569f95 in ap_rgetline_core ()
#10 0x000055fb6156a73d in ap_get_mime_headers_core ()
#11 0x000055fb6156d5a4 in ap_read_request ()
#12 0x000055fb61597355 in ?? ()
#13 0x000055fb6158d7f8 in ap_run_process_connection ()
#14 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so

Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202391559 in OPENSSL_init_crypto () from target:/lib64/libcrypto.so.1.1
#1  0x00007fa202369506 in ERR_get_state () from target:/lib64/libcrypto.so.1.1
#2  0x00007fa2023697d5 in ?? () from target:/lib64/libcrypto.so.1.1
#3  0x00007fa202525f3e in SSL_get_error () from target:/lib64/libssl.so.1.1
#4  0x00007fa2025a045a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#5  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#6  0x000055fb61569f95 in ap_rgetline_core ()
#7  0x000055fb6156a73d in ap_get_mime_headers_core ()
#8  0x000055fb6156d5a4 in ap_read_request ()
#9  0x000055fb61597355 in ?? ()
#10 0x000055fb6158d7f8 in ap_run_process_connection ()
#11 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#12 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#13 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#14 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6

Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202eacc7f in malloc () from target:/lib64/libc.so.6
#1  0x00007fa20251170c in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa202510f36 in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#6  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#7  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#8  0x000055fb61569f95 in ap_rgetline_core ()
#9  0x000055fb6156a73d in ap_get_mime_headers_core ()
#10 0x000055fb6156d5a4 in ap_read_request ()
#11 0x000055fb61597355 in ?? ()
#12 0x000055fb6158d7f8 in ap_run_process_connection ()
#13 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#14 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202eaaf01 in _int_malloc () from target:/lib64/libc.so.6
#1  0x00007fa202eacc8a in malloc () from target:/lib64/libc.so.6
#2  0x00007fa20251170c in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202510f36 in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#6  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#7  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#8  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#9  0x000055fb61569f95 in ap_rgetline_core ()
#10 0x000055fb6156a73d in ap_get_mime_headers_core ()
#11 0x000055fb6156d5a4 in ap_read_request ()
#12 0x000055fb61597355 in ?? ()
#13 0x000055fb6158d7f8 in ap_run_process_connection ()
#14 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa20250ef88 in ?? () from target:/lib64/libssl.so.1.1
#1  0x00007fa2025131ce in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa202510b7e in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#6  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#7  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#8  0x000055fb61569f95 in ap_rgetline_core ()
#9  0x000055fb6156a73d in ap_get_mime_headers_core ()
#10 0x000055fb6156d5a4 in ap_read_request ()
#11 0x000055fb61597355 in ?? ()
#12 0x000055fb6158d7f8 in ap_run_process_connection ()
#13 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#14 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202ea9707 in _int_free () from target:/lib64/libc.so.6
#1  0x00007fa202511980 in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa20250f0de in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa2025131ce in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202510b7e in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#6  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#7  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#8  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#9  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#10 0x000055fb61569f95 in ap_rgetline_core ()
#11 0x000055fb6156a73d in ap_get_mime_headers_core ()
#12 0x000055fb6156d5a4 in ap_read_request ()
#13 0x000055fb61597355 in ?? ()
#14 0x000055fb6158d7f8 in ap_run_process_connection ()
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202ea9db8 in _int_free () from target:/lib64/libc.so.6
#1  0x00007fa202511980 in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa20250f0de in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa2025131ce in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202510b7e in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#6  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#7  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#8  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#9  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#10 0x000055fb61569f95 in ap_rgetline_core ()
#11 0x000055fb6156a73d in ap_get_mime_headers_core ()
#12 0x000055fb6156d5a4 in ap_read_request ()
#13 0x000055fb61597355 in ?? ()
#14 0x000055fb6158d7f8 in ap_run_process_connection ()
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa2030021c0 in pthread_once () from target:/lib64/libpthread.so.0
#1  0x00007fa2023f94cd in CRYPTO_THREAD_run_once () from target:/lib64/libcrypto.so.1.1
#2  0x00007fa202391576 in OPENSSL_init_crypto () from target:/lib64/libcrypto.so.1.1
#3  0x00007fa202369506 in ERR_get_state () from target:/lib64/libcrypto.so.1.1
#4  0x00007fa2023697d5 in ?? () from target:/lib64/libcrypto.so.1.1
#5  0x00007fa202525f3e in SSL_get_error () from target:/lib64/libssl.so.1.1
#6  0x00007fa2025a045a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#7  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#8  0x000055fb61569f95 in ap_rgetline_core ()
#9  0x000055fb6156a73d in ap_get_mime_headers_core ()
#10 0x000055fb6156d5a4 in ap_read_request ()
#11 0x000055fb61597355 in ?? ()
#12 0x000055fb6158d7f8 in ap_run_process_connection ()
#13 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#14 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202510b31 in ?? () from target:/lib64/libssl.so.1.1
#1  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#4  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#5  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#6  0x000055fb61569f95 in ap_rgetline_core ()
#7  0x000055fb6156a73d in ap_get_mime_headers_core ()
#8  0x000055fb6156d5a4 in ap_read_request ()
#9  0x000055fb61597355 in ?? ()
#10 0x000055fb6158d7f8 in ap_run_process_connection ()
#11 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#12 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#13 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#14 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202509020 in BIO_read@plt () from target:/lib64/libssl.so.1.1
#1  0x00007fa20250efff in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa2025131ce in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202510b7e in ?? () from target:/lib64/libssl.so.1.1
#4  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#5  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#6  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#7  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#8  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#9  0x000055fb61569f95 in ap_rgetline_core ()
#10 0x000055fb6156a73d in ap_get_mime_headers_core ()
#11 0x000055fb6156d5a4 in ap_read_request ()
#12 0x000055fb61597355 in ?? ()
#13 0x000055fb6158d7f8 in ap_run_process_connection ()
#14 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa20236975b in ERR_clear_error () from target:/lib64/libcrypto.so.1.1
#1  0x00007fa2025a0403 in ?? () from target:/etc/httpd/modules/mod_ssl.so
#2  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#3  0x000055fb61569f95 in ap_rgetline_core ()
#4  0x000055fb6156a73d in ap_get_mime_headers_core ()
#5  0x000055fb6156d5a4 in ap_read_request ()
#6  0x000055fb61597355 in ?? ()
#7  0x000055fb6158d7f8 in ap_run_process_connection ()
#8  0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#9  0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#10 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#11 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6
Thread 59 (Thread 0x7fa1defed700 (LWP 380)):
#0  0x00007fa20300073c in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libpthread.so.0
#1  0x000055fb6159233d in ap_queue_pop_something ()
Thread 60 (Thread 0x7fa1de7ec700 (LWP 381)):
#0  0x00007fa202510b22 in ?? () from target:/lib64/libssl.so.1.1
#1  0x00007fa202517f75 in ?? () from target:/lib64/libssl.so.1.1
#2  0x00007fa202522e4e in ?? () from target:/lib64/libssl.so.1.1
#3  0x00007fa202522f57 in SSL_read () from target:/lib64/libssl.so.1.1
#4  0x00007fa2025a041a in ?? () from target:/etc/httpd/modules/mod_ssl.so
#5  0x00007fa2025a26fc in ?? () from target:/etc/httpd/modules/mod_ssl.so
#6  0x000055fb61569f95 in ap_rgetline_core ()
#7  0x000055fb6156a73d in ap_get_mime_headers_core ()
#8  0x000055fb6156d5a4 in ap_read_request ()
#9  0x000055fb61597355 in ?? ()
#10 0x000055fb6158d7f8 in ap_run_process_connection ()
#11 0x00007fa202677f97 in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#12 0x00007fa20267893c in ?? () from target:/etc/httpd/modules/mod_mpm_event.so
#13 0x00007fa202ffa58e in start_thread () from target:/lib64/libpthread.so.0
#14 0x00007fa202f236a3 in clone () from target:/lib64/libc.so.6
Comment 1 mike bayer 2019-01-02 01:19:11 UTC 
OK yup a kind soul on twitter pointed me to the source of these requests and it is https://www.ssllabs.com/ssltest/analyze.html, I hit my server with this and it's that same IP number 64.41.200.103 and it reliably reproduces the process hanging at 100% CPU when the series of tests gets to about 92%.    let me know if you need more information.
Comment 2 Joe Orton 2019-01-02 09:20:39 UTC 
Nice analysis, thanks.  What OpenSSL version?  Also can you work out what thread 60 is doing, is it spinning inside OPENSSL_init_crypto() ?
Comment 3 mike bayer 2019-01-02 15:21:50 UTC 
this is fedora 29 so packages look like:

openssl-1.1.1-3.fc29.x86_64
openssl-pkcs11-0.4.8-2.fc29.x86_64
compat-openssl10-1.0.2o-3.fc29.x86_64
openssl-libs-1.1.1-3.fc29.x86_64
httpd-tools-2.4.37-5.fc29.x86_64
httpd-2.4.37-5.fc29.x86_64
httpd-filesystem-2.4.37-5.fc29.noarch


as for what thread 60 is doing, I'm not versed at the moment in stepping through C code with gdb, I would instead hope that this issue is easily reproducible by developers?      E.g. create any SSL setup with the above libraries and event MPM (which I have a feeling is not even necessary) and then hit your server with https://www.ssllabs.com/ssltest/analyze.html.    Works every time here and per the linked discussion other people are seeing it as well.
Comment 4 mike bayer 2019-01-02 15:27:01 UTC 
From my end this kind of looks like a pretty big DOS vulnerability, anyone can just run the attacks from that publicly available online tool a few dozen times against any site running the latest Apache and bring it down.
Comment 5 Ruediger Pluem 2019-01-07 07:29:07 UTC 
(In reply to mike bayer from comment #3)
> this is fedora 29 so packages look like:
> 
> openssl-1.1.1-3.fc29.x86_64
> openssl-pkcs11-0.4.8-2.fc29.x86_64
> compat-openssl10-1.0.2o-3.fc29.x86_64
> openssl-libs-1.1.1-3.fc29.x86_64
> httpd-tools-2.4.37-5.fc29.x86_64
> httpd-2.4.37-5.fc29.x86_64
> httpd-filesystem-2.4.37-5.fc29.noarch

Might be related to openssl-1.1.1. I checked a self build 2.4.37 build against RedHat 7's openssl-1.0.2k and there is no spinning. Could this be related to TLS 1.3? Or to the API changes in 1.1.1 that have special handling in the code?
Comment 6 William A. Rowe Jr. 2019-01-08 08:15:27 UTC 
This sounds familiar, there are discussions of mod_ssl/openssl 1.1.1 compatibility
on the mailing list, related specifically to callback handling. Since those were
solved in 2.4.37, contemporaneous to openssl release 1.1.1a (which did not ship with
FC29), we may be of limited help. I have not observed the behavior you observe with
that specific combination, and have returned to qualsys scanner on many occasions.

It may be worth raising a fedora bug on this, and point back to this ticket, since
both your httpd 2.4 and openssl 1.1.1 packages are forked. 

It might also be specific to your (default Fedora?) configuration, would you mind 
sharing that here (or on a corresponding fedora ticket?)
Comment 7 mike bayer 2019-01-08 17:40:43 UTC 
fedora issue with the conf for this vhost is opened at https://bugzilla.redhat.com/show_bug.cgi?id=1664414
Comment 8 William A. Rowe Jr. 2019-01-08 21:26:12 UTC 
This may be based on a misunderstanding by our developers of the SSL_clear_error() function, as first identified here;

https://bz.apache.org/bugzilla/show_bug.cgi?id=62590
Comment 9 Joe Orton 2019-01-10 15:46:21 UTC 
Fixed in r1850946.
Comment 10 mike bayer 2019-01-14 16:15:42 UTC 
so... what's the timeline for this to be released and getting it downstream at least as a downloadable rpm?  I'm being hit with this issue daily.   also any thoughts on why this issue is not more widespread?
Comment 11 William A. Rowe Jr. 2019-01-15 01:09:42 UTC 
Proposed in httpd-2.4/STATUS for backport.
Comment 12 William A. Rowe Jr. 2019-01-16 17:24:56 UTC 
Committed to branches/2.4.x/ in r1851471 for inclusion in the next
2.4.38 release candidate.
Comment 13 Joe Orton 2019-01-17 13:33:40 UTC 
*** Bug 63083 has been marked as a duplicate of this bug. ***