Bug 54905 - Apache2 loops in child Segfaults
Summary: Apache2 loops in child Segfaults
Status: NEEDINFO
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Core (show other bugs)
Version: 2.4.4
Hardware: PC Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-04-29 14:30 UTC by Roy
Modified: 2014-01-20 00:25 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Roy 2013-04-29 14:30:17 UTC
After a random time (sometimes on startup, sometimes later) Apache 2.4.4 gets into a segfault loop (child processes are faulting and dying, while parent attempts to recreate child that are faulting again).
I've tried it on various machines - with no success.
I have tried disabling all irrelevant modules (I'm not using php, and anyway it happens when not serving any requests!)

Attached is the stack-trace from gdb, from all threads.

(gdb) thread apply all bt

Thread 28 (LWP 4568):
#0  0x00007f3cc2bc8707 in kill () from /lib/x86_64-linux-gnu/libc.so.6
#1  <signal handler called>
#2  0x00007f3cc38168d2 in ?? () from /lib64/ld-linux-x86-64.so.2
#3  0x00007f3cc3806be4 in ?? () from /lib64/ld-linux-x86-64.so.2
#4  0x00007f3cc381184b in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f3cc380d176 in ?? () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f3cc381131a in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f3cc2cc2612 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x00007f3cc380d176 in ?? () from /lib64/ld-linux-x86-64.so.2
#9  0x00007f3cc2cc26d4 in __libc_dlopen_mode () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007f3cc2f613bc in pthread_cancel_init () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007f3cc2f6157c in _Unwind_ForcedUnwind () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007f3cc2f5f6a0 in __pthread_unwind () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007f3cc2f59ff5 in pthread_exit () from /lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007f3cc31933b6 in apr_thread_exit (thd=<optimized out>, retval=<optimized out>) at threadproc/unix/thread.c:207
#15 0x0000000000465f27 in start_threads (thd=0xa26b48, dummy=0xa26ac8) at event.c:2001
#16 0x00007f3cc2f58e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#17 0x00007f3cc2c85ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7f3cadfeb700 (LWP 4594)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cadfeb700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7f3cae7ec700 (LWP 4593)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cae7ec700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7f3caefed700 (LWP 4592)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3caefed700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7f3caf7ee700 (LWP 4591)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3caf7ee700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 23 (LWP 4590):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3caffef700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 22 (LWP 4589):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cbe80e700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7f3cb0ff1700 (LWP 4588)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb07f0700 in ?? ()
---Type <return> to continue, or q <return> to quit---
#3  0x0000000000000000 in ?? ()

Thread 20 (LWP 4587):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb0ff1700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 19 (LWP 4586):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb3ff7700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 18 (LWP 4585):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb1ff3700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 17 (LWP 4584):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb2ff5700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f3cbe80e700 (LWP 4569)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb37f6700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 15 (LWP 4583):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb17f2700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 14 (LWP 4582):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb47f8700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 13 (LWP 4581):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb27f4700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f3cbe00d700 (LWP 4570)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb4ff9700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 11 (LWP 4580):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb57fa700 in ?? ()
#3  0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 10 (LWP 4579):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb5ffb700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 9 (LWP 4578):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb5ffb700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f3cbd80c700 (LWP 4571)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cbd80c700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 7 (LWP 4577):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb67fc700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f3cbd00b700 (LWP 4572)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb7fff700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 5 (LWP 4576):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cbc80a700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 4 (LWP 4575):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cbd00b700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f3cb7fff700 (LWP 4574)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb6ffd700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f3cbc80a700 (LWP 4573)):
#0  0x00007f3cc2c85c91 in clone () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3cc2f58dc0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f3cb77fe700 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f3cc3a12700 (LWP 4544)):
#0  0x00007f3cc2f5fd2d in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000000000467764 in read (__nbytes=1, __buf=0x7fff7ad826ef, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/unistd.h:45
#2  ap_event_pod_check (pod=<optimized out>) at pod.c:58
#3  0x0000000000427f1f in child_main (child_num_arg=4) at event.c:2185
#4  0x0000000000465b37 in make_child (s=0x951508, slot=4) at event.c:2272
#5  0x00000000004667d2 in perform_idle_server_maintenance () at event.c:2469
#6  server_main_loop (remaining_children_to_start=0) at event.c:2597
---Type <return> to continue, or q <return> to quit---
#7  event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:2656
#8  0x000000000042ee96 in ap_run_mpm (pconf=0x928138, plog=0x955378, s=0x951508) at mpm_common.c:98
#9  0x00000000004289b3 in main (argc=3, argv=0x7fff7ad82ac8) at main.c:777
Comment 1 Roy 2013-04-29 15:07:14 UTC
The error log looks something like this:

[Mon Apr 29 16:33:01.153356 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18240 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.334155 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18242 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.335361 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18264 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.335485 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18326 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.335662 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18327 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.335806 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18329 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.335883 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18384 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.335943 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18411 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.336000 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18412 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.336056 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18447 exit signal Segmentation fault (11)
[Mon Apr 29 16:33:02.336112 2013] [core:notice] [pid 1885:tid 139898956883712] AH00052: child pid 18524 exit signal Segmentation fault (11)


and goes on and on...
Comment 2 Rainer Jung 2013-05-04 11:44:49 UTC
Sounds to me like a platform bug. Which Linux distribution is it exactly? Can you check with your distributor?

apr_thread_exit() simply calls pthread_exit(NULL). IMHO nothing to fix here.

Could it be a low memory situation on the system, so that it kills the processes?

Regards,

Rainer
Comment 3 Stefan Fritsch 2013-05-05 18:16:52 UTC
Doing 'bt full' for the thread that crashes could give further hints if there is a problem in httpd. If there is a package with debug symbols for libc, installing that beforehand may be a good idea, too. For example, on Debian, this is called libc6-dbg. Is there anything else interesting in the error log? Try "grep -v AH00052 ..."