We see a lot of SIGBUS crashes with 2.4.54 lately. We can't reproduce it in lab – it's our customers' production sites, so we can't just take their data or experiment with their setup either. It's a Gentoo based custom Linux system and worker MPM. Core was generated by `/usr/sbin/httpd -DFOREGROUND -f /etc/apache/https/httpd.conf'. Program terminated with signal SIGBUS, Bus error. #0 __memmove_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:348 348 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory. [Current thread is 1 (Thread 0x7f9476822640 (LWP 34724))] (gdb) bt full #0 __memmove_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:348 No locals. #1 0x00007f948770f34b in memcpy (__len=32768, __src=<optimized out>, __dest=<optimized out>) at /usr/include/bits/string_fortified.h:29 No locals. #2 read_buf (strm=0x7f9284087cd8, buf=<optimized out>, size=size@entry=32768) at /home/build/zlib-1.2.13/deflate.c:1227 len = 32768 #3 0x00007f948770f3fb in fill_window (s=s@entry=0x7f9284000b60) at /home/build/zlib-1.2.13/deflate.c:1581 n = <optimized out> more = 32768 wsize = 32768 #4 0x00007f948770f8c2 in deflate_slow (s=0x7f9284000b60, flush=0) at /home/build/zlib-1.2.13/deflate.c:1994 hash_head = <optimized out> bflush = <optimized out> #5 0x00007f948771108e in deflate (strm=strm@entry=0x7f9284087cd8, flush=flush@entry=0) at /home/build/zlib-1.2.13/deflate.c:1057 bstate = <optimized out> old_flush = <optimized out> s = 0x7f9284000b60 #6 0x0000556c9f2cefed in deflate_out_filter (f=<optimized out>, bb=0x7f9284087c90) at mod_deflate.c:987 b = <optimized out> e = 0x7f92840ad188 r = 0x7f92840526e0 ctx = 0x7f9284087cd8 zRC = <optimized out> len = 306843 blen = 306843 data = 0x7f9486109000 "" c = <optimized out> #7 0x0000556c9f2cb66a in filter_harness (f=0x7f9284087ac0, bb=0x7f9284087c90) at mod_filter.c:323 ret = <optimized out> cachecontrol = <optimized out> ctx = 0x7f9284087b10 filter = <optimized out> #8 0x00007f948658b37a in output_filter (f=0x7f9284087a90, bb_in=<optimized out>) at apache2_io.c:1054 r = <optimized out> msr = 0x7f9284054678 bucket = <optimized out> eos_bucket = <optimized out> rc = <optimized out> start_skipping = <optimized out> #9 0x0000556c9f29687e in default_handler (r=0x7f92840526e0) at core.c:4997 c = 0x7f928407c5e0 bb = 0x7f9284087c90 e = <optimized out> d = 0x7f928404eb38 errstatus = <optimized out> fd = 0x7f9284087b28 status = <optimized out> bld_content_md5 = <optimized out> #10 0x0000556c9f2a6240 in ap_run_handler (r=r@entry=0x7f92840526e0) at config.c:170 pHook = <optimized out> n = 14 rv = -1 #11 0x0000556c9f2a8df6 in ap_invoke_handler (r=r@entry=0x7f92840526e0) at config.c:448 handler = <optimized out> p = <optimized out> result = 0 old_handler = 0x0 ignore = <optimized out> #12 0x0000556c9f2d7ae8 in ap_process_async_request (r=r@entry=0x7f92840526e0) at http_request.c:452 c = 0x7f928407c5e0 access_status = 0 #13 0x0000556c9f2d7d1f in ap_process_request (r=r@entry=0x7f92840526e0) at http_request.c:487 bb = 0x7f92840526e0 b = <optimized out> c = 0x7f928407c5e0 rv = <optimized out> #14 0x00007f948664c475 in h2_task_process_request (c=0x7f928407c5e0, task=<optimized out>) at h2_task.c:671 req = <optimized out> cs = <optimized out> r = 0x7f92840526e0 req = <optimized out> cs = <optimized out> r = <optimized out> #15 h2_task_process_conn (c=0x7f928407c5e0) at h2_task.c:713 --Type <RET> for more, q to quit, c to continue without paging-- ctx = <optimized out> ctx = <optimized out> #16 h2_task_process_conn (c=0x7f928407c5e0) at h2_task.c:700 ctx = <optimized out> #17 0x0000556c9f2b2580 in ap_run_process_connection (c=c@entry=0x7f928407c5e0) at connection.c:42 pHook = <optimized out> n = 1 rv = -1 #18 0x00007f948664d1d2 in h2_task_do (task=0x7f92840a90e0, thread=thread@entry=0x556c9f9759f0, worker_id=<optimized out>) at h2_task.c:631 c = 0x7f928407c5e0 #19 0x00007f9486650cca in slot_run (thread=0x556c9f9759f0, wctx=0x556c9f8ce080) at h2_workers.c:263 slot = 0x556c9f8ce080 #20 0x00007f9487417777 in start_thread (arg=<optimized out>) at pthread_create.c:435 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140275620128320, 9186818040044405033, 140275620128320, 0, 140275901101296, 0, -9198937240569429719, -9198834409542041303}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = { prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #21 0x00007f948749a3d0 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 No locals. (gdb)
Do you have MMAP enabled and do you modify content in-place wile it is being served? https://httpd.apache.org/docs/2.4/mod/core.html#enablemmap
I had the same backtrace, and turning off EnableMMAP seems to have fixed it. It was challenging to find out exactly which vhost was causing it, because I didn't want to turn it off globally. It would have been nice to have more info in the logs without having to resort to an httpd build with debug symbols plus gdb to find out.
It is good practice not to modify content in-place for an active web server, otherwise you will risk corrupted responses even if you disable mmap.