After a random time (sometimes on startup, sometimes later) Apache 2.4.4 gets into a segfault loop (child processes are faulting and dying, while parent attempts to recreate child that are faulting again). I've tried it on various machines - with no success. I have tried disabling all irrelevant modules (I'm not using php, and anyway it happens when not serving any requests!) Attached is the stack-trace from gdb, from all threads. (gdb) thread apply all bt Thread 28 (LWP 4568): #0 0x00007f3cc2bc8707 in kill () from /lib/x86_64-linux-gnu/libc.so.6 #1 <signal handler called> #2 0x00007f3cc38168d2 in ?? () from /lib64/ld-linux-x86-64.so.2 #3 0x00007f3cc3806be4 in ?? () from /lib64/ld-linux-x86-64.so.2 #4 0x00007f3cc381184b in ?? () from /lib64/ld-linux-x86-64.so.2 #5 0x00007f3cc380d176 in ?? () from /lib64/ld-linux-x86-64.so.2 #6 0x00007f3cc381131a in ?? () from /lib64/ld-linux-x86-64.so.2 #7 0x00007f3cc2cc2612 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x00007f3cc380d176 in ?? () from /lib64/ld-linux-x86-64.so.2 #9 0x00007f3cc2cc26d4 in __libc_dlopen_mode () from /lib/x86_64-linux-gnu/libc.so.6 #10 0x00007f3cc2f613bc in pthread_cancel_init () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x00007f3cc2f6157c in _Unwind_ForcedUnwind () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f3cc2f5f6a0 in __pthread_unwind () from /lib/x86_64-linux-gnu/libpthread.so.0 #13 0x00007f3cc2f59ff5 in pthread_exit () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007f3cc31933b6 in apr_thread_exit (thd=<optimized out>, retval=<optimized out>) at threadproc/unix/thread.c:207 #15 0x0000000000465f27 in start_threads (thd=0xa26b48, dummy=0xa26ac8) at event.c:2001 #16 0x00007f3cc2f58e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #17 0x00007f3cc2c85ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #18 0x0000000000000000 in ?? () Thread 27 (Thread 0x7f3cadfeb700 (LWP 4594)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cadfeb700 in ?? () #3 0x0000000000000000 in ?? () Thread 26 (Thread 0x7f3cae7ec700 (LWP 4593)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cae7ec700 in ?? () #3 0x0000000000000000 in ?? () Thread 25 (Thread 0x7f3caefed700 (LWP 4592)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3caefed700 in ?? () #3 0x0000000000000000 in ?? () Thread 24 (Thread 0x7f3caf7ee700 (LWP 4591)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3caf7ee700 in ?? () #3 0x0000000000000000 in ?? () Thread 23 (LWP 4590): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3caffef700 in ?? () #3 0x0000000000000000 in ?? () Thread 22 (LWP 4589): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cbe80e700 in ?? () #3 0x0000000000000000 in ?? () Thread 21 (Thread 0x7f3cb0ff1700 (LWP 4588)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb07f0700 in ?? () ---Type <return> to continue, or q <return> to quit--- #3 0x0000000000000000 in ?? () Thread 20 (LWP 4587): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb0ff1700 in ?? () #3 0x0000000000000000 in ?? () Thread 19 (LWP 4586): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb3ff7700 in ?? () #3 0x0000000000000000 in ?? () Thread 18 (LWP 4585): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb1ff3700 in ?? () #3 0x0000000000000000 in ?? () Thread 17 (LWP 4584): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb2ff5700 in ?? () #3 0x0000000000000000 in ?? () Thread 16 (Thread 0x7f3cbe80e700 (LWP 4569)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb37f6700 in ?? () #3 0x0000000000000000 in ?? () Thread 15 (LWP 4583): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb17f2700 in ?? () #3 0x0000000000000000 in ?? () Thread 14 (LWP 4582): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb47f8700 in ?? () #3 0x0000000000000000 in ?? () Thread 13 (LWP 4581): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb27f4700 in ?? () #3 0x0000000000000000 in ?? () Thread 12 (Thread 0x7f3cbe00d700 (LWP 4570)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb4ff9700 in ?? () #3 0x0000000000000000 in ?? () Thread 11 (LWP 4580): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb57fa700 in ?? () #3 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- Thread 10 (LWP 4579): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb5ffb700 in ?? () #3 0x0000000000000000 in ?? () Thread 9 (LWP 4578): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb5ffb700 in ?? () #3 0x0000000000000000 in ?? () Thread 8 (Thread 0x7f3cbd80c700 (LWP 4571)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cbd80c700 in ?? () #3 0x0000000000000000 in ?? () Thread 7 (LWP 4577): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb67fc700 in ?? () #3 0x0000000000000000 in ?? () Thread 6 (Thread 0x7f3cbd00b700 (LWP 4572)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb7fff700 in ?? () #3 0x0000000000000000 in ?? () Thread 5 (LWP 4576): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cbc80a700 in ?? () #3 0x0000000000000000 in ?? () Thread 4 (LWP 4575): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cbd00b700 in ?? () #3 0x0000000000000000 in ?? () Thread 3 (Thread 0x7f3cb7fff700 (LWP 4574)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb6ffd700 in ?? () #3 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f3cbc80a700 (LWP 4573)): #0 0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3cb77fe700 in ?? () #3 0x0000000000000000 in ?? () Thread 1 (Thread 0x7f3cc3a12700 (LWP 4544)): #0 0x00007f3cc2f5fd2d in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000467764 in read (__nbytes=1, __buf=0x7fff7ad826ef, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/unistd.h:45 #2 ap_event_pod_check (pod=<optimized out>) at pod.c:58 #3 0x0000000000427f1f in child_main (child_num_arg=4) at event.c:2185 #4 0x0000000000465b37 in make_child (s=0x951508, slot=4) at event.c:2272 #5 0x00000000004667d2 in perform_idle_server_maintenance () at event.c:2469 #6 server_main_loop (remaining_children_to_start=0) at event.c:2597 ---Type <return> to continue, or q <return> to quit--- #7 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:2656 #8 0x000000000042ee96 in ap_run_mpm (pconf=0x928138, plog=0x955378, s=0x951508) at mpm_common.c:98 #9 0x00000000004289b3 in main (argc=3, argv=0x7fff7ad82ac8) at main.c:777
The error log looks something like this: [Mon Apr 29 16:33:01.153356 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18240 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.334155 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18242 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.335361 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18264 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.335485 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18326 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.335662 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18327 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.335806 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18329 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.335883 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18384 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.335943 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18411 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.336000 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18412 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.336056 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18447 exit signal Segmentation fault (11) [Mon Apr 29 16:33:02.336112 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18524 exit signal Segmentation fault (11) and goes on and on...
Sounds to me like a platform bug. Which Linux distribution is it exactly? Can you check with your distributor? apr_thread_exit() simply calls pthread_exit(NULL). IMHO nothing to fix here. Could it be a low memory situation on the system, so that it kills the processes? Regards, Rainer
Doing 'bt full' for the thread that crashes could give further hints if there is a problem in httpd. If there is a package with debug symbols for libc, installing that beforehand may be a good idea, too. For example, on Debian, this is called libc6-dbg. Is there anything else interesting in the error log? Try "grep -v AH00052 ..."