We are observing a few deadlocking issues around process recycle time (one example is below, but we are seeing others). As an operational mitigation we would like to be able to use either the GracefullTimeout parameter or a new parameter to indicate how long we are willing to wait for the threads to shutdown aka a deadlock timeout, before going into an ungraceful termination of threads. requests_this_child was at -1 in each case (and we have MaxRequetsPerChild set) I have read the source for 2.2.22 and 2.4.4 for the mpm_worker and the issue will exist in all versions (as the listener thread will instruct threads to terminate, perform its own shutdown then exit, the main thread will eventually hit pthread_join and block. We are also looking at changing the openssl version as we believe this is the source of the deadlocking. However a operational mitigation against any deadlock is still desirable. On platforms that support it pthread_tryjoin_np could perhaps be used ... timer_create may be more portable. This may mean the listener needs to be the last to die or pass the pod fds other to the mainthread so it can perform a ungraceful shutdown. (gdb) info threads Id Target Id Frame 2 Thread 0x7f4c57fff700 (LWP 6028) "apache2" 0x00007f4c7c282be4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x7f4c7cd3b760 (LWP 6008) "apache2" 0x00007f4c7c27d1f8 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0 (gdb) thread 2 [Switching to thread 2 (Thread 0x7f4c57fff700 (LWP 6028))] #0 0x00007f4c7c282be4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 (gdb) bt #0 0x00007f4c7c282be4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f4c7c27e1e5 in _L_lock_883 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f4c7c27e03a in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007f4c7c4ad890 in apr_thread_mutex_lock () from /usr/lib/libapr-1.so.0 #4 0x00007f4c78469443 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007f4c78469d9c in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007f4c7846ac81 in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #7 0x00007f4c7846c369 in ERR_set_mark () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #8 0x00007f4c784597fa in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #9 0x00007f4c7846e598 in EVP_CipherInit_ex () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #10 0x00007f4c78772f46 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #11 0x00007f4c7876ab7e in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #12 0x00007f4c7876b819 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #13 0x00007f4c7876cb90 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #14 0x00007f4c78760054 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #15 0x00007f4c78761990 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #16 0x00007f4c7876ec13 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0 #17 0x00007f4c789b01a0 in ?? () from /usr/lib/apache2/modules/mod_ssl.so #18 0x00007f4c789b0740 in ?? () from /usr/lib/apache2/modules/mod_ssl.so #19 0x00007f4c7cd7ec86 in ap_rgetline_core (s=0x7f4c6c67a0d0, n=8192, read=0x7f4c57ffed40, r=0x7f4c6c67a0a0, fold=0, bb=0x7f4c6c67b7e8) at /build/buildd/apache2-2.2.20/server/protocol.c:231 #20 0x00007f4c7cd80c2e in read_request_line (bb=0x7f4c6c67b7e8, r=0x7f4c6c67a0a0) at /build/buildd/apache2-2.2.20/server/protocol.c:596 #21 ap_read_request (conn=0x7f4c6c6a92b8) at /build/buildd/apache2-2.2.20/server/protocol.c:921 #22 0x00007f4c7cd9a520 in ap_process_http_connection (c=0x7f4c6c6a92b8) at /build/buildd/apache2-2.2.20/modules/http/http_core.c:183 #23 0x00007f4c7cd944b8 in ap_run_process_connection (c=0x7f4c6c6a92b8) at /build/buildd/apache2-2.2.20/server/connection.c:43 #24 0x00007f4c7cda38ac in process_socket (bucket_alloc=<optimized out>, my_thread_num=28, my_child_num=11, sock=0x7f4c6c6a90a0, p=<optimized out>) at /build/buildd/apache2-2.2.20/server/mpm/worker/worker.c:544 #25 worker_thread (thd=0x7f4c7cbfc980, dummy=<optimized out>) at /build/buildd/apache2-2.2.20/server/mpm/worker/worker.c:894 #26 0x00007f4c7c4beeb3 in ?? () from /usr/lib/libapr-1.so.0 #27 0x00007f4c7c27befc in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #28 0x00007f4c7bfb5f8d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #29 0x0000000000000000 in ?? ()
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.