MPM: Event OS: Amazon Linux AMI release 2011.09 (CentOS) APR: 1.4.6 APR-UTIL: 1.4.1 Our system is 1 sandbox server, 2 test and 3 prod. It has occured once (one test server). 2.4.1 in use less than a week. Servers are clones. (Actually based on the server in mind.) Child finally killed with -5. Post compiled and installed mod_geoip2_1.2.7 + 4 of my own modules. $ sudo strace -ff -F -s200 -o /tmp/strace.out -p 27203 $ sort /tmp/strace.out.272* epoll_wait(85, {}, 25, 100) = 0 epoll_wait(85, {}, 25, 100) = 0 epoll_wait(85, {}, 25, 100) = 0 ...more of same futex(0x17a65bc, FUTEX_WAIT_PRIVATE, 41, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0x17a65bc, FUTEX_WAIT_PRIVATE, 43, NULL) = -1 EAGAIN (Resource temporarily unavailable) futex(0x17a65bc, FUTEX_WAIT_PRIVATE, 45, NULL) = -1 EAGAIN (Resource temporarily unavailable) ...more of same futex(0x17a65bc, FUTEX_WAIT_PRIVATE, 88, NULL futex(0x17a65bc, FUTEX_WAIT_PRIVATE, 88, NULL futex(0x17a65bc, FUTEX_WAIT_PRIVATE, 88, NULL ...more of same futex(0x7fa11bfe79d0, FUTEX_WAIT, 27284, NULL gettimeofday({1332018558, 639020}, NULL) = 0 gettimeofday({1332018560, 550880}, NULL) = 0 gettimeofday({1332018560, 550985}, NULL) = 0 ...more of same $ /usr/local/apache2/bin/apachectl -V Server version: Apache/2.4.1 (Unix) Server built: Mar 9 2012 18:36:08 Server's Module Magic Number: 20120211:0 Server loaded: APR 1.4.6, APR-UTIL 1.4.1 Compiled using: APR 1.4.6, APR-UTIL 1.4.1 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/usr/local/apache2" -D SUEXEC_BIN="/usr/local/apache2/bin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" $ cat config.nice #! /bin/sh # # Created by configure "./configure" \ "--enable-sed" \ "--enable-usertrack" \ "--enable-unique-id" \ "--enable-proxy" \ "--enable-unixd" \ "--enable-rewrite" \ "--with-mpm=event" \ "--with-included-apr" \ "--enable-deflate" \ "--enable-ssl" \ "--enable-log-debug" \ "--enable-log-forensic" \ "--enable-logio" \ "$@"
A stack trace would be much more useful than strace output. Use $ gdb -p <pid of looping process> (gdb) thread apply all bt full
I'll do that if it happens again.
For completeness: If you had mod_dumpio and DumpIOInput enabled at the time of the problem, this is likely a duplicate of PR 52914.
I had not.
It happened again, here's three gdb screens: No symbol table info available. #1 0x0000000000466ad5 in ap_queue_pop_something (queue=0x9ae530, sd=0x7f0336ffce78, ecs=0x7f0336ffce70, p=0x7f0336ffce68, te_out=0x7f0336ffce60) at fdqueue.c:438 elem = <value optimized out> rv = <value optimized out> #2 0x00000000004644ad in worker_thread (thd=0x9af0a8, dummy=<value optimized out>) at event.c:1702 ti = <value optimized out> process_slot = 0 thread_slot = 5 csd = 0x7f0338037600 cs = 0x7f0338037808 ptrans = 0x7f0338037578 rv = <value optimized out> is_idle = 1 te = 0x0 #3 0x00007f03451e27f1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f0344f1f92d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 21 (Thread 0x7f03367fc700 (LWP 4357)): #0 0x00007f03451e63dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000000000466ad5 in ap_queue_pop_something (queue=0x9ae530, sd=0x7f03367fbe78, ecs=0x7f03367fbe70, p=0x7f03367fbe68, te_out=0x7f03367fbe60) at fdqueue.c:438 elem = <value optimized out> rv = <value optimized out> #2 0x00000000004644ad in worker_thread (thd=0x9af0d8, dummy=<value optimized out>) at event.c:1702 ti = <value optimized out> process_slot = 0 thread_slot = 6 csd = 0x7f03380396d0 cs = 0x0 ptrans = 0x7f0338039658 rv = <value optimized out> is_idle = 1 te = 0x0 #3 0x00007f03451e27f1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f0344f1f92d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 20 (Thread 0x7f0335ffb700 (LWP 4358)): #0 0x00007f03451e63dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000000000466ad5 in ap_queue_pop_something (queue=0x9ae530, sd=0x7f0335ffae78, ecs=0x7f0335ffae70, p=0x7f0335ffae68, te_out=0x7f0335ffae60) at fdqueue.c:438 elem = <value optimized out> rv = <value optimized out> #2 0x00000000004644ad in worker_thread (thd=0x9af108, dummy=<value optimized out>) at event.c:1702 ti = <value optimized out> process_slot = 0 thread_slot = 7 csd = 0x7f0338037600 cs = 0x7f0338037808 ptrans = 0x7f0338037578 rv = <value optimized out> ---Type <return> to continue, or q <return> to quit--- is_idle = 1 te = 0x0 #3 0x00007f03451e27f1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f0344f1f92d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 19 (Thread 0x7f03357fa700 (LWP 4359)): #0 apr_file_write (thefile=0x7f031001af78, buf=<value optimized out>, nbytes=<value optimized out>) at file_io/unix/readwrite.c:176 pos = <value optimized out> blocksize = 0 size = 68 rv = 0 #1 0x00007f0340b37fb4 in savefiltermethod (f=0x7f0310006a58, bb=0x7f0310007eb8) at ufmodules/mod_savecontent.c:128 ctxt = 0x7f0310007ed8 bytes = 68 buff = 0x7f0310019002 "<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"no\" lang=\"no\">\n<head>\n<title>Panelet Plug</title>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" /> \n\n<style type=\"text/css\">\n "... b = 0x7f0310087a18 rv = <value optimized out> #2 0x00007f03433a7c8a in filter_harness (f=0x7f0310006a58, bb=0x7f0310007eb8) at mod_filter.c:313 ret = <value optimized out> cachecontrol = <value optimized out> str = <value optimized out> ctx = 0x7f0310006af0 filter = <value optimized out> #3 0x00007f0340d3ad0a in substitute_filter (f=0x7f0310006a30, bb=0x7f0310007e10) at ufmodules/mod_ouf_substitute.c:566 bytes = 0 len = <value optimized out> fbytes = 139651759166952 buff = 0x7f0310019046 "<head>\n<title>Panelet Plug</title>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" /> \n\n<style type=\"text/css\">\n \n body {\n \n font-family: Arial, Tahoma, Verdana;"... nl = <value optimized out> bflat = 0x6b <Address 0x6b out of bounds> b = <value optimized out> tmp_b = <value optimized out> tmp_bb = 0x0 rv = <value optimized out> ctx = 0x7f0310007e30 #4 0x00007f03433a7c8a in filter_harness (f=0x7f0310006a30, bb=0x7f0310007e10) at mod_filter.c:313 ret = <value optimized out> cachecontrol = <value optimized out> str = <value optimized out> ctx = 0x7f0310006ad8 filter = <value optimized out> #5 0x00007f03431a4a0c in substitute_filter (f=0x7f0310006a08, bb=0x7f0310007d90) at mod_substitute.c:511 bytes = 0 len = <value optimized out> fbytes = 139651130075384 buff = 0x7f03100199a0 " <div class=\"question\"><a href=\"http://politisk.tv2.no//spesial/panelet/sporsmaal/h" nl = <value optimized out> bflat = 0x1fa0 <Address 0x1fa0 out of bounds> b = <value optimized out> tmp_b = <value optimized out> ---Type <return> to continue, or q <return> to quit--- tmp_bb = 0x0 rv = <value optimized out> ctx = 0x7f0310004898 #6 0x00007f03433a7c8a in filter_harness (f=0x7f0310006a08, bb=0x7f0310007d90) at mod_filter.c:313 ret = <value optimized out> cachecontrol = <value optimized out> str = <value optimized out> ctx = 0x7f0310006ac0 filter = <value optimized out> #7 0x00007f0340933115 in inflate_out_filter (f=0x7f03100069e0, bb=0x7f0310007630) at mod_deflate.c:1316 rv = <value optimized out> data = 0x114630b "" b = <value optimized out> len = 1093 zlib_method = <value optimized out> zlib_flags = <value optimized out> e = 0x7f0310087b58 r = 0x7f031000aa50 ctx = 0x7f0310007ce0 zRC = <value optimized out> rv = <value optimized out> c = 0x98e478 #8 0x00007f03433a7c8a in filter_harness (f=0x7f03100069e0, bb=0x7f0310007630) at mod_filter.c:313 ret = <value optimized out> cachecontrol = <value optimized out> str = <value optimized out> ctx = 0x7f0310006aa8 filter = <value optimized out> #9 0x00007f0341d68599 in ap_proxy_http_process_response (p=0x7f031000a9d8, r=<value optimized out>, backend_ptr=0x7f03357f9b18, worker=0xa6c4f8, conf=0xa6c268, server_portstr=0x7f03357f9af0 "") at mod_proxy_http.c:1973 readbytes = 1103 rv = <value optimized out> mode = <value optimized out> finish = 0 rc = <value optimized out> c = 0x7f0338039950 buffer = "\000\nntent-Encoding\000 gzip\000\nser-Agent\000\n\000\n8\000\n\000\000\000\000!\000\000\000\000\000\000\000\001\000\000\000!", '\000' <repeats 11 times>, "\001\000\000\000,{\177\065\003\177", '\000' <repeats 14 times>, "\005\000\000\000\350\220\177\065\003\177\000\000\232\020\335\361\001\000\000\000@\204\226;\000\000\000\000\360\220\177\065\003\177\000\000ӈ\177\065\003\177\000\000\350\220\177\065\003\177\000\000\f\221\177\065\003\177\000\000\060{\177\065\003\177\000\000${\177\065\003\177\000\000\b\221\177\065\003\177\000\000\020{\177\065\003\177\000\000\001\000\000\000\000\b\000\000({\177\065\003\177\000\000 L`t", '\000' <repeats 12 times>"\370, z\177\065\003\177"... buf = <value optimized out> keepchar = <value optimized out> e = <value optimized out> bb = 0x7f0310007630 tmp_bb = 0x7f0310007678 pass_bb = 0x7f0310007650 len = 15 backasswards = <value optimized out> interim_response = 0 pread_len = <value optimized out> save_table = <value optimized out> backend_broke = 0 hop_by_hop_hdrs = {0x7f0341d6a716 "Keep-Alive", 0x7f0341d6a9c7 "Proxy-Authenticate", 0x7f0341d6a721 "TE", 0x7f0341d6a724 "Trailer", 0x7f0341d6a72c "Upgrade", 0x0} i = <value optimized out> ---Type <return> to continue, or q <return> to quit---
#1 0x00007f0340b37fb4 in savefiltermethod (f=0x7f0310006a58, bb=0x7f0310007eb8) at ufmodules/mod_savecontent.c:128 Likely a third-party module bug.