Over the past years Apache has crashed several times every hour. I've been running Apache 2 since 2.0.40 and now we're up at 2.0.61 and still experiencing these crashes so I guess it's about time I report this bug. Our site heavily relies on SSI, in particular <!--#include virtual=..--> (they're often nested aswell), and mod_rewrite. Output is compressed with mod_deflate. Often the crashdumps looks like this: crashdump[26541]: crashdump[26541]: Thread 20 Crashed: crashdump[26541]: 0 httpd 0x000047a0 bndm + 120 (mod_include.c:317) crashdump[26541]: 1 httpd 0x0000a100 find_start_sequence + 136 (mod_include.c:2388) crashdump[26541]: 2 httpd 0x0000bad0 send_parsed_content + 1288 (mod_include.c:3054) crashdump[26541]: 3 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[26541]: 4 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[26541]: 5 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[26541]: 6 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[26541]: 7 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[26541]: 8 httpd 0x00051f98 ap_run_sub_req + 104 (request.c:1855) crashdump[26541]: 9 httpd 0x00005a68 handle_include + 968 (mod_include.c:782) crashdump[26541]: 10 httpd 0x0000cc80 send_parsed_content + 5816 (mod_include.c:3309) crashdump[26541]: 11 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[26541]: 12 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[26541]: 13 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[26541]: 14 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[26541]: 15 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[26541]: 16 httpd 0x0001d980 ap_process_request + 148 (http_request.c:249) crashdump[26541]: 17 httpd 0x000145a4 ap_process_http_connection + 136 (http_core.c:255) crashdump[26541]: 18 httpd 0x0006bcdc ap_run_process_connection + 136 (connection.c:43) crashdump[26541]: 19 httpd 0x0006c318 ap_process_connection + 132 (connection.c:178) crashdump[26541]: 20 httpd 0x0003b408 process_socket + 196 (worker.c:523) crashdump[26541]: 21 httpd 0x0003befc worker_thread + 552 (worker.c:843) crashdump[26541]: 22 libapr-0.0.dylib 0x003796f0 dummy_worker + 68 (thread.c:105) crashdump[26541]: 23 libSystem.B.dylib 0x9002bd08 _pthread_body + 96 -------------------------- crashdump[4103]: crashdump[4103]: Thread 15 Crashed: crashdump[4103]: 0 httpd 0x000047a0 bndm + 120 (mod_include.c:317) crashdump[4103]: 1 httpd 0x0000a100 find_start_sequence + 136 (mod_include.c:2388) crashdump[4103]: 2 httpd 0x0000bad0 send_parsed_content + 1288 (mod_include.c:3054) crashdump[4103]: 3 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[4103]: 4 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[4103]: 5 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[4103]: 6 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[4103]: 7 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[4103]: 8 httpd 0x00051f98 ap_run_sub_req + 104 (request.c:1855) crashdump[4103]: 9 httpd 0x00005a68 handle_include + 968 (mod_include.c:782) crashdump[4103]: 10 httpd 0x0000cc80 send_parsed_content + 5816 (mod_include.c:3309) crashdump[4103]: 11 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[4103]: 12 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[4103]: 13 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[4103]: 14 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[4103]: 15 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[4103]: 16 httpd 0x0001d980 ap_process_request + 148 (http_request.c:249) crashdump[4103]: 17 httpd 0x000145a4 ap_process_http_connection + 136 (http_core.c:255) crashdump[4103]: 18 httpd 0x0006bcdc ap_run_process_connection + 136 (connection.c:43) crashdump[4103]: 19 httpd 0x0006c318 ap_process_connection + 132 (connection.c:178) crashdump[4103]: 20 httpd 0x0003b408 process_socket + 196 (worker.c:523) crashdump[4103]: 21 httpd 0x0003befc worker_thread + 552 (worker.c:843) crashdump[4103]: 22 libapr-0.0.dylib 0x003796f0 dummy_worker + 68 (thread.c:105) crashdump[4103]: 23 libSystem.B.dylib 0x9002bd08 _pthread_body + 96 ------------------ I saw someone else having a similar problem here. There was an open bug about random crashes in crc32 and bndm, but with Apache 2.2. I've tried to log the PID along with the HTTP requests into a logfile and grepping out the PID after the crash but I guess the thread crashes before the log is written since I'm not finding anything interesting there. I've not been able to reproduce this but it keeps happening several times each hour. Since we've got a fair share of traffic to this host I can't really gdb Apache and wait for a crash and see what's going on. It might be a memory corruption since we're getting crashes in other places aswell. It's not likely to be a hardware problem since the box have been replaced two times already (for performance reasons) and the crashes have continued on the new hardware aswell. Here's one of the other crashes we're getting quite frequently but not as frequently as with bndm(): crashdump[14604]: crashdump[14604]: Thread 7 Crashed: crashdump[14604]: 0 httpd 0x0000a6c0 find_directive + 284 (mod_include.c:2552) crashdump[14604]: 1 httpd 0x0000bf14 send_parsed_content + 2380 (mod_include.c:3118) crashdump[14604]: 2 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[14604]: 3 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[14604]: 4 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[14604]: 5 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[14604]: 6 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[14604]: 7 httpd 0x00051f98 ap_run_sub_req + 104 (request.c:1855) crashdump[14604]: 8 httpd 0x00005a68 handle_include + 968 (mod_include.c:782) crashdump[14604]: 9 httpd 0x0000cc80 send_parsed_content + 5816 (mod_include.c:3309) crashdump[14604]: 10 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[14604]: 11 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[14604]: 12 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[14604]: 13 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[14604]: 14 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[14604]: 15 httpd 0x00051f98 ap_run_sub_req + 104 (request.c:1855) crashdump[14604]: 16 httpd 0x00005a68 handle_include + 968 (mod_include.c:782) crashdump[14604]: 17 httpd 0x0000cc80 send_parsed_content + 5816 (mod_include.c:3309) crashdump[14604]: 18 httpd 0x0000dbf8 includes_filter + 1468 (mod_include.c:3591) crashdump[14604]: 19 httpd 0x00068e30 ap_pass_brigade + 236 (util_filter.c:512) crashdump[14604]: 20 httpd 0x0004a5c8 default_handler + 1888 (core.c:3648) crashdump[14604]: 21 httpd 0x00060038 ap_run_handler + 136 (config.c:152) crashdump[14604]: 22 httpd 0x00060d8c ap_invoke_handler + 424 (config.c:364) crashdump[14604]: 23 httpd 0x0001d980 ap_process_request + 148 (http_request.c:249) crashdump[14604]: 24 httpd 0x000145a4 ap_process_http_connection + 136 (http_core.c:255) crashdump[14604]: 25 httpd 0x0006bcdc ap_run_process_connection + 136 (connection.c:43) crashdump[14604]: 26 httpd 0x0006c318 ap_process_connection + 132 (connection.c:178) crashdump[14604]: 27 httpd 0x0003b408 process_socket + 196 (worker.c:523) crashdump[14604]: 28 httpd 0x0003befc worker_thread + 552 (worker.c:843) crashdump[14604]: 29 libapr-0.0.dylib 0x003796f0 dummy_worker + 68 (thread.c:105) ------ This problem has occured on Mac OS X 10.3 and up to 10.4.10, both Server versions. We've been running a broad list of different versions of Apache 2, from .40 to .61. This problem occurs with the threaded worker. Does anyone have any idea what's going on here?
I recently worked with a user to track down an issue with identical backtraces - the root cause was PR 36780; (optimistically?) marking as a duplicate. *** This bug has been marked as a duplicate of 36780 ***
Reporter says the patch for bug 36780 didn't help, so re-opening. Noah: 1) can you reproduce this with prefork rather than worker? 2) can you get a coredump and do some debugging with gdb? What is needed out of the core dump is: # gdb /path/to/httpd core.dump (gdb) source /patch/to/httpd/src/.gdbinit then select a frame with a apr_bucket_brigade * in scope, e.g. send_parsed_content, e.g. "up 2" from that first "Thread 20", and do: (gdb) dump_brigade bb to dump the contents of the brigade.
(In reply to comment #2) > Reporter says the patch for bug 36780 didn't help, so re-opening. > > Noah: > > 1) can you reproduce this with prefork rather than worker? I'll try. Is it crucial for the debugging part or just interesting to know whether it might be a threading related problem or not? > 2) can you get a coredump and do some debugging with gdb? > > What is needed out of the core dump is: > > # gdb /path/to/httpd core.dump > (gdb) source /patch/to/httpd/src/.gdbinit > > then select a frame with a apr_bucket_brigade * in scope, e.g. > send_parsed_content, e.g. "up 2" from that first "Thread 20", and do: > > (gdb) dump_brigade bb > > to dump the contents of the brigade. I've enabled coredumps and I'll see if I can find some time to look into it.
This is what Mac OS X's crashdump process have to say. Thread 2 Crashed: 0 httpd 0x00004760 bndm + 120 (mod_include.c:317) 1 httpd 0x0000a0c0 find_start_sequence + 136 (mod_include.c:2388) 2 httpd 0x0000ba90 send_parsed_content + 1288 (mod_include.c:3054) 3 httpd 0x0000dbb8 includes_filter + 1468 (mod_include.c:3591) 4 httpd 0x00068e20 ap_pass_brigade + 236 (util_filter.c:512) 5 httpd 0x0004a588 default_handler + 1888 (core.c:3648) 6 httpd 0x00060028 ap_run_handler + 136 (config.c:152) 7 httpd 0x00060d7c ap_invoke_handler + 424 (config.c:364) 8 httpd 0x00051f58 ap_run_sub_req + 104 (request.c:1855) 9 httpd 0x00005a28 handle_include + 968 (mod_include.c:782) 10 httpd 0x0000cc40 send_parsed_content + 5816 (mod_include.c:3309) 11 httpd 0x0000dbb8 includes_filter + 1468 (mod_include.c:3591) 12 httpd 0x00068e20 ap_pass_brigade + 236 (util_filter.c:512) 13 httpd 0x0004a588 default_handler + 1888 (core.c:3648) 14 httpd 0x00060028 ap_run_handler + 136 (config.c:152) 15 httpd 0x00060d7c ap_invoke_handler + 424 (config.c:364) 16 httpd 0x0001d940 ap_process_request + 148 (http_request.c:249) 17 httpd 0x00014564 ap_process_http_connection + 136 (http_core.c:255) 18 httpd 0x0006bcdc ap_run_process_connection + 136 (connection.c:43) 19 httpd 0x0006c318 ap_process_connection + 132 (connection.c:178) 20 httpd 0x0003b3c8 process_socket + 196 (worker.c:523) 21 httpd 0x0003bebc worker_thread + 552 (worker.c:843) 22 libapr-0.0.dylib 0x003796f0 dummy_worker + 68 (thread.c:105) 23 libSystem.B.dylib 0x9002bd08 _pthread_body + 96 Now, if I'm running GDB, I'm getting this. # gdb -q /opt/FrontA/apache/bin/httpd core.9148 Reading symbols for shared libraries ........... done #0 0x900148a8 in read () (gdb) source /data/FrontA/software/httpd-2.0.61/.gdbinit (gdb) thread 2 [Switching to thread 2 (core thread 1)] #0 0x9001f88c in select () (gdb) bt #0 0x9001f88c in select () #1 0x003851ec in ?? () #2 0x00386728 in ?? () #3 0x00372e2c in ?? () #4 0x001af150 in ?? () #5 0x001aba20 in ?? () #6 0x0004ac48 in core_input_filter (f=0x133d060, b=0x11d4bd0, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3802 #7 0x00068d04 in ap_get_brigade (next=0x133d060, bb=0x11d4bd0, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:475 #8 0x0004a83c in net_time_filter (f=0x137ff70, b=0x11d4bd0, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3705 #9 0x00068d04 in ap_get_brigade (next=0x137ff70, bb=0x11d4bd0, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:475 #10 0x0005b51c in ap_rgetline_core (s=0x1327c58, n=8192, read=0xf0101b40, r=0x1327c40, fold=0, bb=0x11d4bd0) at protocol.c:230 #11 0x0005beac in read_request_line (r=0x1327c40, bb=0x11d4bd0) at protocol.c:586 #12 0x0005cb70 in ap_read_request (conn=0x133ce10) at protocol.c:877 #13 0x00014648 in ap_process_http_connection () at apr_xml.c:737 #14 0x0006bcdc in ap_run_process_connection (c=0x133ce10) at connection.c:43 #15 0x0006c318 in ap_process_connection (c=0x133ce10, csd=0x1321200) at connection.c:176 #16 0x0003b3c8 in process_socket (p=0x11973a0, sock=0x1321200, my_child_num=7, my_thread_num=0, bucket_alloc=0x185fc18) at worker.c:522 #17 0x0003bebc in worker_thread (thd=0x416c60, dummy=0x469760) at worker.c:842 #18 0x003796f0 in ?? () #19 0x9002bd08 in _pthread_body () GDB and crashdump sees different addresses for "0x00014648 in ap_process_http_connection". Is this normal? After that it seems that the stackframes differs wildly, at least in my eyes. gdb -v says GNU gdb 6.1-20040303 (Apple version gdb-384) (Mon Mar 21 00:05:26 GMT 2005) Apache was compiled using: ./configure --enable-so --enable-mods-shared=auth-anon auth-digest file-cache cache disk-cache mem-cache deflate mime-magic expires headers use rtrack unique-id proxy proxy-http ssl vhost-alias cgi info speling rewrite --with-mpm=prefork --mandir=/usr/share/man --prefix=/opt/FrontA/apache-2 .0.61-prefork --enable-pool-debug Five years back or so when I last used GDB I recall it had issues with threads, is that still so? Maybe I should try to see if the problem occurs with MPM prefork unless you've got some input on this thing? I tried doing a bunch of backtraces on other core files aswell but their backtraces all look like the one above.
You need to have #include virtual on file which changes during request. I guess as file is mmapped and if it is not complete or changes during request you get these crashes. Reproducible. RHEL4 x86_64 Stock 2.2.10, compiled: CFLAGS='-O0 -g' ./configure --with-mpm=prefork --enable-maintainer-mode Create shtml file with include virtual, so that it includes complex shtml-file. # cat html/test-43533.shtml <html><body> <!--#include virtual="inc-43533.shtml" --> </body></html> # cat html/inc-43533-orig.shtml <!--#config timefmt="%M" --> <!--#if expr="$date_local > 00 && $date_local < 10" --> <!--#include virtual="/foo2.html" --> <!--#endif --> <!--#if expr="$date_local > 09 && $date_local < 20" --> <!--#include virtual="/foo2.html" --> <!--#endif --> <!--#if expr="$date_local > 19 && $date_local < 30" --> <!--#include virtual="/foo2.html" --> <!--#endif --> <!--#if expr="$date_local > 29 && $date_local < 40" --> <!--#include virtual="/foo2.html" --> <!--#endif --> <!--#if expr="$date_local > 39 && $date_local < 50" --> <!--#include virtual="/foo2.html" --> <!--#endif --> <!--#if expr="$date_local > 49 && $date_local < 60" --> <!--#include virtual="/foo2.html" --> <!--#endif --> Create job that changes file continuously: # (while sleep 1;do cp html/inc-43533-orig.shtml html/inc-43533.shtml;done)& Start prefork (or worker) apache with -X option. Request test-43533.shtml with ab and wait it to crash. Below several points. Program received signal SIGBUS, Bus error. [Switching to Thread 182897648736 (LWP 13275)] 0x0000000000450d71 in find_directive (ctx=0x65f7a8, data=0x2a959131de "if expr=\"$date_local > 39 && $date_local < 50\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n <!--#include virtual=\"/foo2.html\" --"..., len=217, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:2726 2726 while (p < ep && !apr_isspace(*p)) { (gdb) where #0 0x0000000000450d71 in find_directive (ctx=0x65f7a8, data=0x2a959131de "if expr=\"$date_local > 39 && $date_local < 50\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n <!--#include virtual=\"/foo2.html\" --"..., len=217, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:2726 #1 0x0000000000452068 in send_parsed_content (f=0x65f5e8, bb=0x65f768) at mod_include.c:3300 #2 0x0000000000453075 in includes_filter (f=0x65f5e8, b=0x65f768) at mod_include.c:3651 #3 0x00000000004490c4 in ap_pass_brigade (next=0x65f5e8, bb=0x65f768) at util_filter.c:526 #4 0x0000000000434b44 in default_handler (r=0x65da78) at core.c:3740 #5 0x000000000043bc88 in ap_run_handler (r=0x65da78) at config.c:157 #6 0x000000000043c52f in ap_invoke_handler (r=0x65da78) at config.c:372 #7 0x0000000000439128 in ap_run_sub_req (r=0x65da78) at request.c:1876 #8 0x000000000044e027 in handle_include (ctx=0x654128, f=0x653e90, bb=0x654c40) at mod_include.c:1737 #9 0x00000000004527b8 in send_parsed_content (f=0x653e90, bb=0x6540e8) at mod_include.c:3432 #10 0x0000000000453075 in includes_filter (f=0x653e90, b=0x6540e8) at mod_include.c:3651 #11 0x00000000004490c4 in ap_pass_brigade (next=0x653e90, bb=0x6540e8) at util_filter.c:526 #12 0x0000000000434b44 in default_handler (r=0x64fa08) at core.c:3740 #13 0x000000000043bc88 in ap_run_handler (r=0x64fa08) at config.c:157 #14 0x000000000043c52f in ap_invoke_handler (r=0x64fa08) at config.c:372 #15 0x000000000045bd5f in ap_process_request (r=0x64fa08) at http_request.c:258 #16 0x0000000000458f84 in ap_process_http_connection (c=0x64bbd8) at http_core.c:190 #17 0x0000000000444d25 in ap_run_process_connection (c=0x64bbd8) at connection.c:43 #18 0x0000000000445156 in ap_process_connection (c=0x64bbd8, csd=0x64b9e8) at connection.c:178 #19 0x00000000004747a6 in child_main (child_num_arg=0) at prefork.c:650 #20 0x0000000000474876 in make_child (s=0x5b9160, slot=0) at prefork.c:690 #21 0x0000000000474e0a in ap_mpm_run (_pconf=0x5b0138, plog=0x5f2348, s=0x5b9160) at prefork.c:966 #22 0x0000000000423bea in main (argc=6, argv=0x7fbffff7d8) at main.c:740 Program received signal SIGBUS, Bus error. [Switching to Thread 182897648736 (LWP 15546)] 0x00000000004508d4 in bndm (t=0x6559b8, h=0x2a95913247 "\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n", hl=112) at mod_include.c:2520 2520 d &= T[(unsigned char) *p--]; #0 0x00000000004508d4 in bndm (t=0x6559b8, h=0x2a95913247 "\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n", hl=112) at mod_include.c:2520 #1 0x00000000004509bf in find_start_sequence (ctx=0x659778, data=0x2a95913247 "\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n", len=112) at mod_include.c:2561 #2 0x0000000000451d20 in send_parsed_content (f=0x6595b8, bb=0x659738) at mod_include.c:3238 #3 0x0000000000453075 in includes_filter (f=0x6595b8, b=0x659738) at mod_include.c:3651 ... Program received signal SIGBUS, Bus error. [Switching to Thread 182897648736 (LWP 16265)] 0x0000000000450d71 in find_directive (ctx=0x65b788, data=0x2a959131de "", len=217, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:2726 2726 while (p < ep && !apr_isspace(*p)) { Program received signal SIGBUS, Bus error. [Switching to Thread 182897648736 (LWP 16695)] 0x000000000045102b in find_arg_or_tail (ctx=0x6637c8, data=0x2a959130e7 " -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 19 && $date_local < 30\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 29 && $date_local < 40\" -->\n <!--#includ"..., len=464) at mod_include.c:2823 2823 while (p < ep && apr_isspace(*p)) { Program received signal SIGBUS, Bus error. [Switching to Thread 182897648736 (LWP 17474)] 0x0000003f31072584 in memcpy () from /lib64/tls/libc.so.6 (gdb) where #0 0x0000003f31072584 in memcpy () from /lib64/tls/libc.so.6 #1 0x0000002a955731ff in apr_brigade_flatten (bb=0x657830, c=0x65baa8 "/foo2.html", len=0x7fbfffed98) at buckets/apr_brigade.c:252 #2 0x0000002a955732a6 in apr_brigade_pflatten (bb=0x657830, c=0x65ba88, len=0x65ba90, pool=0x65b9e8) at buckets/apr_brigade.c:294 #3 0x00000000004523e1 in send_parsed_content (f=0x6575a8, bb=0x657728) at mod_include.c:3370 ... Program received signal SIGBUS, Bus error. [Switching to Thread 182897648736 (LWP 17906)] 0x00000000004516ca in find_argument (ctx=0x659778, data=0x2a9591306d "/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 09 && $date_local < 20\" -->\n <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 19 && $date_local < 30\" -->\n "..., len=586, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:3028 3028 if (intern->quote && *p == '\\') { (gdb) print *(*b->list->next)->type $16 = {name = 0x2a9558b054 "MMAP", num_func = 5, is_metadata = APR_BUCKET_DATA, destroy = 0x2a95575042 <mmap_bucket_destroy>, read = 0x2a95574f98 <mmap_bucket_read>, setaside = 0x2a9557519a <mmap_bucket_setaside>, split = 0x422698, copy = 0x421cb8}
(In reply to comment #5) > You need to have #include virtual on file which changes during request. I guess > as file is mmapped and if it is not complete or changes during request you get > these crashes. Nice catch. It could very well be just that. We do have frequent updates of those files (via FTP) on the server and they're rewritten each time they're updated, not moved in place.