Bug 43533

Summary: Frequent crashes in mod_include's bndm()
Product: Apache httpd-2 Reporter: Noah Williamsson <noah>
Component: mod_includeAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: REOPENED ---    
Severity: normal CC: Markus.Linnala
Priority: P2    
Version: 2.0.61   
Target Milestone: ---   
Hardware: Macintosh   
OS: Mac OS X 10.4   

Description Noah Williamsson 2007-10-02 03:54:41 UTC
Over the past years Apache has crashed several times every hour.
I've been running Apache 2 since 2.0.40 and now we're up at 2.0.61 and still
experiencing these crashes so I guess it's about time I report this bug.

Our site heavily relies on SSI, in particular <!--#include virtual=..-->
(they're often nested aswell), and mod_rewrite. Output is compressed with
mod_deflate. Often the crashdumps looks like this:

crashdump[26541]: crashdump[26541]: Thread 20 Crashed:
crashdump[26541]: 0   httpd                      0x000047a0 bndm + 120
(mod_include.c:317)
crashdump[26541]: 1   httpd                      0x0000a100 find_start_sequence
+ 136 (mod_include.c:2388)
crashdump[26541]: 2   httpd                      0x0000bad0 send_parsed_content
+ 1288 (mod_include.c:3054)
crashdump[26541]: 3   httpd                      0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[26541]: 4   httpd                      0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[26541]: 5   httpd                      0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[26541]: 6   httpd                      0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[26541]: 7   httpd                      0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[26541]: 8   httpd                      0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[26541]: 9   httpd                      0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[26541]: 10  httpd                      0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[26541]: 11  httpd                      0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[26541]: 12  httpd                      0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[26541]: 13  httpd                      0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[26541]: 14  httpd                      0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[26541]: 15  httpd                      0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[26541]: 16  httpd                      0x0001d980 ap_process_request +
148 (http_request.c:249)
crashdump[26541]: 17  httpd                      0x000145a4
ap_process_http_connection + 136 (http_core.c:255)
crashdump[26541]: 18  httpd                      0x0006bcdc
ap_run_process_connection + 136 (connection.c:43)
crashdump[26541]: 19  httpd                      0x0006c318
ap_process_connection + 132 (connection.c:178)
crashdump[26541]: 20  httpd                      0x0003b408 process_socket + 196
(worker.c:523)
crashdump[26541]: 21  httpd                      0x0003befc worker_thread + 552
(worker.c:843)
crashdump[26541]: 22  libapr-0.0.dylib           0x003796f0 dummy_worker + 68
(thread.c:105)
crashdump[26541]: 23  libSystem.B.dylib          0x9002bd08 _pthread_body + 96
--------------------------
crashdump[4103]: crashdump[4103]: Thread 15 Crashed:
crashdump[4103]: 0   httpd                       0x000047a0 bndm + 120
(mod_include.c:317)
crashdump[4103]: 1   httpd                       0x0000a100 find_start_sequence
+ 136 (mod_include.c:2388)
crashdump[4103]: 2   httpd                       0x0000bad0 send_parsed_content
+ 1288 (mod_include.c:3054)
crashdump[4103]: 3   httpd                       0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[4103]: 4   httpd                       0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[4103]: 5   httpd                       0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[4103]: 6   httpd                       0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[4103]: 7   httpd                       0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[4103]: 8   httpd                       0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[4103]: 9   httpd                       0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[4103]: 10  httpd                       0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[4103]: 11  httpd                       0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[4103]: 12  httpd                       0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[4103]: 13  httpd                       0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[4103]: 14  httpd                       0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[4103]: 15  httpd                       0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[4103]: 16  httpd                       0x0001d980 ap_process_request +
148 (http_request.c:249)
crashdump[4103]: 17  httpd                       0x000145a4
ap_process_http_connection + 136 (http_core.c:255)
crashdump[4103]: 18  httpd                       0x0006bcdc
ap_run_process_connection + 136 (connection.c:43)
crashdump[4103]: 19  httpd                       0x0006c318
ap_process_connection + 132 (connection.c:178)
crashdump[4103]: 20  httpd                       0x0003b408 process_socket + 196
(worker.c:523)
crashdump[4103]: 21  httpd                       0x0003befc worker_thread + 552
(worker.c:843)
crashdump[4103]: 22  libapr-0.0.dylib            0x003796f0 dummy_worker + 68
(thread.c:105)
crashdump[4103]: 23  libSystem.B.dylib           0x9002bd08 _pthread_body + 96
------------------


I saw someone else having a similar problem here. There was an open bug about
random crashes in crc32 and bndm, but with Apache 2.2.

I've tried to log the PID along with the HTTP requests into a logfile and
grepping out the PID after the crash but I guess the thread crashes before the
log is written since I'm not finding anything interesting there.

I've not been able to reproduce this but it keeps happening several times each
hour. Since we've got a fair share of traffic to this host I can't really gdb
Apache and wait for a crash and see what's going on. 

It might be a memory corruption since we're getting crashes in other places
aswell. It's not likely to be a hardware problem since the box have been
replaced two times already (for performance reasons) and the crashes have
continued on the new hardware aswell.

Here's one of the other crashes we're getting quite frequently but not as
frequently as with bndm():
crashdump[14604]: crashdump[14604]: Thread 7 Crashed:
crashdump[14604]: 0   httpd                      0x0000a6c0 find_directive + 284
(mod_include.c:2552)
crashdump[14604]: 1   httpd                      0x0000bf14 send_parsed_content
+ 2380 (mod_include.c:3118)
crashdump[14604]: 2   httpd                      0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[14604]: 3   httpd                      0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[14604]: 4   httpd                      0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[14604]: 5   httpd                      0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[14604]: 6   httpd                      0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[14604]: 7   httpd                      0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[14604]: 8   httpd                      0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[14604]: 9   httpd                      0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[14604]: 10  httpd                      0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[14604]: 11  httpd                      0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[14604]: 12  httpd                      0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[14604]: 13  httpd                      0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[14604]: 14  httpd                      0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[14604]: 15  httpd                      0x00051f98 ap_run_sub_req + 104
(request.c:1855)
crashdump[14604]: 16  httpd                      0x00005a68 handle_include + 968
(mod_include.c:782)
crashdump[14604]: 17  httpd                      0x0000cc80 send_parsed_content
+ 5816 (mod_include.c:3309)
crashdump[14604]: 18  httpd                      0x0000dbf8 includes_filter +
1468 (mod_include.c:3591)
crashdump[14604]: 19  httpd                      0x00068e30 ap_pass_brigade +
236 (util_filter.c:512)
crashdump[14604]: 20  httpd                      0x0004a5c8 default_handler +
1888 (core.c:3648)
crashdump[14604]: 21  httpd                      0x00060038 ap_run_handler + 136
(config.c:152)
crashdump[14604]: 22  httpd                      0x00060d8c ap_invoke_handler +
424 (config.c:364)
crashdump[14604]: 23  httpd                      0x0001d980 ap_process_request +
148 (http_request.c:249)
crashdump[14604]: 24  httpd                      0x000145a4
ap_process_http_connection + 136 (http_core.c:255)
crashdump[14604]: 25  httpd                      0x0006bcdc
ap_run_process_connection + 136 (connection.c:43)
crashdump[14604]: 26  httpd                      0x0006c318
ap_process_connection + 132 (connection.c:178)
crashdump[14604]: 27  httpd                      0x0003b408 process_socket + 196
(worker.c:523)
crashdump[14604]: 28  httpd                      0x0003befc worker_thread + 552
(worker.c:843)
crashdump[14604]: 29  libapr-0.0.dylib           0x003796f0 dummy_worker + 68
(thread.c:105)
------


This problem has occured on Mac OS X 10.3 and up to 10.4.10, both Server
versions. We've been running a broad list of different versions of Apache 2,
from .40 to .61.
This problem occurs with the threaded worker.


Does anyone have any idea what's going on here?
Comment 1 Joe Orton 2007-10-11 06:26:53 UTC
I recently worked with a user to track down an issue with identical backtraces -
the root cause was PR 36780; (optimistically?) marking as a duplicate.


*** This bug has been marked as a duplicate of 36780 ***
Comment 2 Joe Orton 2007-10-12 01:28:13 UTC
Reporter says the patch for bug 36780 didn't help, so re-opening.

Noah:

1) can you reproduce this with prefork rather than worker?
2) can you get a coredump and do some debugging with gdb?

What is needed out of the core dump is:

# gdb /path/to/httpd core.dump
(gdb) source /patch/to/httpd/src/.gdbinit

then select a frame with a apr_bucket_brigade * in scope, e.g.
send_parsed_content, e.g. "up 2" from that first "Thread 20", and do:

(gdb) dump_brigade bb

to dump the contents of the brigade.

Comment 3 Noah Williamsson 2007-10-12 02:13:58 UTC
(In reply to comment #2)
> Reporter says the patch for bug 36780 didn't help, so re-opening.
> 
> Noah:
> 
> 1) can you reproduce this with prefork rather than worker?

I'll try.
Is it crucial for the debugging part or just interesting to know whether it
might be a threading related problem or not?


> 2) can you get a coredump and do some debugging with gdb?
> 
> What is needed out of the core dump is:
> 
> # gdb /path/to/httpd core.dump
> (gdb) source /patch/to/httpd/src/.gdbinit
> 
> then select a frame with a apr_bucket_brigade * in scope, e.g.
> send_parsed_content, e.g. "up 2" from that first "Thread 20", and do:
> 
> (gdb) dump_brigade bb
> 
> to dump the contents of the brigade.

I've enabled coredumps and I'll see if I can find some time to look into it.
Comment 4 Noah Williamsson 2007-10-12 15:09:54 UTC
This is what Mac OS X's crashdump process have to say.

Thread 2 Crashed:
0   httpd                       0x00004760 bndm + 120 (mod_include.c:317)
1   httpd                       0x0000a0c0 find_start_sequence + 136
(mod_include.c:2388)
2   httpd                       0x0000ba90 send_parsed_content + 1288
(mod_include.c:3054)
3   httpd                       0x0000dbb8 includes_filter + 1468
(mod_include.c:3591)
4   httpd                       0x00068e20 ap_pass_brigade + 236 (util_filter.c:512)
5   httpd                       0x0004a588 default_handler + 1888 (core.c:3648)
6   httpd                       0x00060028 ap_run_handler + 136 (config.c:152)
7   httpd                       0x00060d7c ap_invoke_handler + 424 (config.c:364)
8   httpd                       0x00051f58 ap_run_sub_req + 104 (request.c:1855)
9   httpd                       0x00005a28 handle_include + 968 (mod_include.c:782)
10  httpd                       0x0000cc40 send_parsed_content + 5816
(mod_include.c:3309)
11  httpd                       0x0000dbb8 includes_filter + 1468
(mod_include.c:3591)
12  httpd                       0x00068e20 ap_pass_brigade + 236 (util_filter.c:512)
13  httpd                       0x0004a588 default_handler + 1888 (core.c:3648)
14  httpd                       0x00060028 ap_run_handler + 136 (config.c:152)
15  httpd                       0x00060d7c ap_invoke_handler + 424 (config.c:364)
16  httpd                       0x0001d940 ap_process_request + 148
(http_request.c:249)
17  httpd                       0x00014564 ap_process_http_connection + 136
(http_core.c:255)
18  httpd                       0x0006bcdc ap_run_process_connection + 136
(connection.c:43)
19  httpd                       0x0006c318 ap_process_connection + 132
(connection.c:178)
20  httpd                       0x0003b3c8 process_socket + 196 (worker.c:523)
21  httpd                       0x0003bebc worker_thread + 552 (worker.c:843)
22  libapr-0.0.dylib            0x003796f0 dummy_worker + 68 (thread.c:105)
23  libSystem.B.dylib           0x9002bd08 _pthread_body + 96



Now, if I'm running GDB, I'm getting this.

# gdb -q /opt/FrontA/apache/bin/httpd core.9148 
Reading symbols for shared libraries ........... done
#0  0x900148a8 in read ()
(gdb) source /data/FrontA/software/httpd-2.0.61/.gdbinit
(gdb) thread 2
[Switching to thread 2 (core thread 1)]
#0  0x9001f88c in select ()
(gdb) bt
#0  0x9001f88c in select ()
#1  0x003851ec in ?? ()
#2  0x00386728 in ?? ()
#3  0x00372e2c in ?? ()
#4  0x001af150 in ?? ()
#5  0x001aba20 in ?? ()
#6  0x0004ac48 in core_input_filter (f=0x133d060, b=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3802
#7  0x00068d04 in ap_get_brigade (next=0x133d060, bb=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:475
#8  0x0004a83c in net_time_filter (f=0x137ff70, b=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3705
#9  0x00068d04 in ap_get_brigade (next=0x137ff70, bb=0x11d4bd0,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:475
#10 0x0005b51c in ap_rgetline_core (s=0x1327c58, n=8192, read=0xf0101b40,
r=0x1327c40, fold=0, bb=0x11d4bd0) at protocol.c:230
#11 0x0005beac in read_request_line (r=0x1327c40, bb=0x11d4bd0) at protocol.c:586
#12 0x0005cb70 in ap_read_request (conn=0x133ce10) at protocol.c:877
#13 0x00014648 in ap_process_http_connection () at apr_xml.c:737
#14 0x0006bcdc in ap_run_process_connection (c=0x133ce10) at connection.c:43
#15 0x0006c318 in ap_process_connection (c=0x133ce10, csd=0x1321200) at
connection.c:176
#16 0x0003b3c8 in process_socket (p=0x11973a0, sock=0x1321200, my_child_num=7,
my_thread_num=0, bucket_alloc=0x185fc18) at worker.c:522
#17 0x0003bebc in worker_thread (thd=0x416c60, dummy=0x469760) at worker.c:842
#18 0x003796f0 in ?? ()
#19 0x9002bd08 in _pthread_body ()


GDB and crashdump sees different addresses for "0x00014648 in
ap_process_http_connection". Is this normal? 
After that it seems that the stackframes differs wildly, at least in my eyes.


gdb -v says
GNU gdb 6.1-20040303 (Apple version gdb-384) (Mon Mar 21 00:05:26 GMT 2005)

Apache was compiled using:
./configure --enable-so --enable-mods-shared=auth-anon auth-digest file-cache
cache disk-cache mem-cache deflate mime-magic expires headers use
rtrack unique-id proxy proxy-http ssl vhost-alias cgi info speling rewrite
--with-mpm=prefork --mandir=/usr/share/man --prefix=/opt/FrontA/apache-2
.0.61-prefork --enable-pool-debug


Five years back or so when I last used GDB I recall it had issues with threads,
is that still so?

Maybe I should try to see if the problem occurs with MPM prefork unless you've
got some input on this thing?

I tried doing a bunch of backtraces on other core files aswell but their
backtraces all look like the one above.
Comment 5 Markus Linnala 2008-11-18 04:02:45 UTC
You need to have #include virtual on file which changes during request. I guess as file is mmapped and if it is not complete or changes during request you get these crashes.

Reproducible. RHEL4 x86_64 Stock 2.2.10, compiled:
CFLAGS='-O0 -g' ./configure --with-mpm=prefork --enable-maintainer-mode

Create shtml file with include virtual, so that it includes complex shtml-file.

# cat html/test-43533.shtml 
<html><body>
<!--#include virtual="inc-43533.shtml" -->
</body></html>
# cat html/inc-43533-orig.shtml 
<!--#config timefmt="%M" -->
<!--#if expr="$date_local > 00 && $date_local < 10" -->
  <!--#include virtual="/foo2.html" -->
<!--#endif -->
<!--#if expr="$date_local > 09 && $date_local < 20" -->
  <!--#include virtual="/foo2.html" -->
<!--#endif -->
<!--#if expr="$date_local > 19 && $date_local < 30" -->
  <!--#include virtual="/foo2.html" -->
<!--#endif -->
<!--#if expr="$date_local > 29 && $date_local < 40" -->
  <!--#include virtual="/foo2.html" -->
<!--#endif -->
<!--#if expr="$date_local > 39 && $date_local < 50" -->
  <!--#include virtual="/foo2.html" -->
<!--#endif -->
<!--#if expr="$date_local > 49 && $date_local < 60" -->
  <!--#include virtual="/foo2.html" -->
<!--#endif -->

Create job that changes file continuously:
# (while sleep 1;do cp html/inc-43533-orig.shtml html/inc-43533.shtml;done)&

Start prefork (or worker) apache with -X option. 

Request test-43533.shtml with ab and wait it to crash.

Below several points.

Program received signal SIGBUS, Bus error.
[Switching to Thread 182897648736 (LWP 13275)]
0x0000000000450d71 in find_directive (ctx=0x65f7a8, 
    data=0x2a959131de "if expr=\"$date_local > 39 && $date_local < 50\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n  <!--#include virtual=\"/foo2.html\" --"..., 
    len=217, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:2726
2726            while (p < ep && !apr_isspace(*p)) {
(gdb) where
#0  0x0000000000450d71 in find_directive (ctx=0x65f7a8, 
    data=0x2a959131de "if expr=\"$date_local > 39 && $date_local < 50\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n  <!--#include virtual=\"/foo2.html\" --"..., 
    len=217, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:2726
#1  0x0000000000452068 in send_parsed_content (f=0x65f5e8, bb=0x65f768) at mod_include.c:3300
#2  0x0000000000453075 in includes_filter (f=0x65f5e8, b=0x65f768) at mod_include.c:3651
#3  0x00000000004490c4 in ap_pass_brigade (next=0x65f5e8, bb=0x65f768) at util_filter.c:526
#4  0x0000000000434b44 in default_handler (r=0x65da78) at core.c:3740
#5  0x000000000043bc88 in ap_run_handler (r=0x65da78) at config.c:157
#6  0x000000000043c52f in ap_invoke_handler (r=0x65da78) at config.c:372
#7  0x0000000000439128 in ap_run_sub_req (r=0x65da78) at request.c:1876
#8  0x000000000044e027 in handle_include (ctx=0x654128, f=0x653e90, bb=0x654c40) at mod_include.c:1737
#9  0x00000000004527b8 in send_parsed_content (f=0x653e90, bb=0x6540e8) at mod_include.c:3432
#10 0x0000000000453075 in includes_filter (f=0x653e90, b=0x6540e8) at mod_include.c:3651
#11 0x00000000004490c4 in ap_pass_brigade (next=0x653e90, bb=0x6540e8) at util_filter.c:526
#12 0x0000000000434b44 in default_handler (r=0x64fa08) at core.c:3740
#13 0x000000000043bc88 in ap_run_handler (r=0x64fa08) at config.c:157
#14 0x000000000043c52f in ap_invoke_handler (r=0x64fa08) at config.c:372
#15 0x000000000045bd5f in ap_process_request (r=0x64fa08) at http_request.c:258
#16 0x0000000000458f84 in ap_process_http_connection (c=0x64bbd8) at http_core.c:190
#17 0x0000000000444d25 in ap_run_process_connection (c=0x64bbd8) at connection.c:43
#18 0x0000000000445156 in ap_process_connection (c=0x64bbd8, csd=0x64b9e8) at connection.c:178
#19 0x00000000004747a6 in child_main (child_num_arg=0) at prefork.c:650
#20 0x0000000000474876 in make_child (s=0x5b9160, slot=0) at prefork.c:690
#21 0x0000000000474e0a in ap_mpm_run (_pconf=0x5b0138, plog=0x5f2348, s=0x5b9160) at prefork.c:966
#22 0x0000000000423bea in main (argc=6, argv=0x7fbffff7d8) at main.c:740


Program received signal SIGBUS, Bus error.
[Switching to Thread 182897648736 (LWP 15546)]
0x00000000004508d4 in bndm (t=0x6559b8, 
    h=0x2a95913247 "\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n", hl=112) at mod_include.c:2520
2520                d &= T[(unsigned char) *p--];
#0  0x00000000004508d4 in bndm (t=0x6559b8, 
    h=0x2a95913247 "\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n", hl=112) at mod_include.c:2520
#1  0x00000000004509bf in find_start_sequence (ctx=0x659778, 
    data=0x2a95913247 "\n<!--#if expr=\"$date_local > 49 && $date_local < 60\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n", len=112) at mod_include.c:2561
#2  0x0000000000451d20 in send_parsed_content (f=0x6595b8, bb=0x659738) at mod_include.c:3238
#3  0x0000000000453075 in includes_filter (f=0x6595b8, b=0x659738) at mod_include.c:3651
...


Program received signal SIGBUS, Bus error.
[Switching to Thread 182897648736 (LWP 16265)]
0x0000000000450d71 in find_directive (ctx=0x65b788, data=0x2a959131de "", len=217, store=0x7fbfffee20, 
    store_len=0x7fbfffee18) at mod_include.c:2726
2726            while (p < ep && !apr_isspace(*p)) {


Program received signal SIGBUS, Bus error.
[Switching to Thread 182897648736 (LWP 16695)]
0x000000000045102b in find_arg_or_tail (ctx=0x6637c8, 
    data=0x2a959130e7 " -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 19 && $date_local < 30\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 29 && $date_local < 40\" -->\n  <!--#includ"..., len=464)
    at mod_include.c:2823
2823        while (p < ep && apr_isspace(*p)) {



Program received signal SIGBUS, Bus error.
[Switching to Thread 182897648736 (LWP 17474)]
0x0000003f31072584 in memcpy () from /lib64/tls/libc.so.6
(gdb) where
#0  0x0000003f31072584 in memcpy () from /lib64/tls/libc.so.6
#1  0x0000002a955731ff in apr_brigade_flatten (bb=0x657830, c=0x65baa8 "/foo2.html", len=0x7fbfffed98)
    at buckets/apr_brigade.c:252
#2  0x0000002a955732a6 in apr_brigade_pflatten (bb=0x657830, c=0x65ba88, len=0x65ba90, pool=0x65b9e8)
    at buckets/apr_brigade.c:294
#3  0x00000000004523e1 in send_parsed_content (f=0x6575a8, bb=0x657728) at mod_include.c:3370
...


Program received signal SIGBUS, Bus error.
[Switching to Thread 182897648736 (LWP 17906)]
0x00000000004516ca in find_argument (ctx=0x659778, 
    data=0x2a9591306d "/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 09 && $date_local < 20\" -->\n  <!--#include virtual=\"/foo2.html\" -->\n<!--#endif -->\n<!--#if expr=\"$date_local > 19 && $date_local < 30\" -->\n  "..., 
    len=586, store=0x7fbfffee20, store_len=0x7fbfffee18) at mod_include.c:3028
3028                if (intern->quote && *p == '\\') {


(gdb) print *(*b->list->next)->type
$16 = {name = 0x2a9558b054 "MMAP", num_func = 5, is_metadata = APR_BUCKET_DATA, 
  destroy = 0x2a95575042 <mmap_bucket_destroy>, read = 0x2a95574f98 <mmap_bucket_read>, 
  setaside = 0x2a9557519a <mmap_bucket_setaside>, split = 0x422698, copy = 0x421cb8}
Comment 6 Noah Williamsson 2008-11-18 04:07:40 UTC
(In reply to comment #5)
> You need to have #include virtual on file which changes during request. I guess
> as file is mmapped and if it is not complete or changes during request you get
> these crashes.

Nice catch. It could very well be just that.
We do have frequent updates of those files (via FTP) on the server and they're rewritten each time they're updated, not moved in place.