Bug 55979 - Segmentation fault
Summary: Segmentation fault
Status: RESOLVED DUPLICATE of bug 50335
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.6
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-01-08 14:54 UTC by Alex Frolkin
Modified: 2014-01-09 09:01 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Frolkin 2014-01-08 14:54:45 UTC
Apache 2.4.6 with libapr 1.4.8 and libaprutil 1.5.2 running a bunch of proxy vhosts (some with proxy balancers) doing SSL encryption and decryption.  Using event MPM, on Linux kernel 3.11.0, 64 bit.

The child processes seem to segfault randomly (I don't know what triggers the segfaults).  Most of the backtraces point to apr_brigade_cleanup.

Here's an example:

#0  0x00007ff8f5e71b07 in apr_brigade_cleanup (data=0x7ff8ec35ca08) at buckets/apr_brigade.c:44
#1  0x00007ff8f5c5348e in run_cleanups (cref=<optimized out>) at ../memory/unix/apr_pools.c:2352
#2  apr_pool_destroy (pool=0x7ff8ec35b028) at ../memory/unix/apr_pools.c:814
#3  0x00007ff8f65429ba in remove_empty_buckets (bb=0x7ff8ec3788f0) at core_filters.c:716
#4  0x00007ff8f6543096 in setaside_remaining_output (f=0x7ff8ec378788, ctx=0x7ff8ec378850, bb=0x7ff8ec3788f0, c=<optimized out>) at core_filters.c:578
#5  0x00007ff8f6543c45 in ap_core_output_filter (f=0x7ff8ec378788, new_bb=0x7ff8ec3788f0) at core_filters.c:562
#6  0x00007ff8f655d86d in ap_process_request_after_handler (r=0x7ff8ec35b0a0) at http_request.c:256
#7  0x00007ff8f655b170 in ap_process_http_async_connection (c=0x7ff8ec378330) at http_core.c:143
#8  ap_process_http_connection (c=0x7ff8ec378330) at http_core.c:228
#9  ap_process_http_connection (c=0x7ff8ec378330) at http_core.c:225
#10 0x00007ff8f65519d8 in ap_run_process_connection (c=0x7ff8ec378330) at connection.c:41
#11 0x00007ff8f277e820 in process_socket (my_thread_num=114, my_child_num=0, cs=0x7ff8ec3782b8, sock=0x7ff8ec3780b0, p=0x7ff8ec378028, thd=0x7ff8f0b87378) at event.c:964
#12 worker_thread (thd=0x7ff8f0b87378, dummy=<optimized out>) at event.c:1812
#13 0x00007ff8f5a24e9a in start_thread (arg=0x7ff8b2794700) at pthread_create.c:308
#14 0x00007ff8f5751cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()

In this case, it seems that the error comes from inside the apr_bucket_delete macro, which (unwrapping the macros) calls APR_RING_NEXT(APR_RING_PREV(e, link), link) (apr-1.4.8/include/apr_ring.h:513), i.e., e->link.prev->link.next.  However:

(gdb) p e->link.prev
$2 = (struct apr_bucket * volatile) 0x0

So that's dereferencing a null pointer.

Interestingly, we don't see any segfaults on a different Apache cluster with identical config (except for the vhosts, and the only differences there are the listening IPs, the proxy backends, and the SSL certificates).  The cluster which exhibits the issue consists of VMware VMs with 2 virtual cores and 6GB RAM, while the cluster where the issue doesn't happen consists of physical machines with 4 cores and 16GB of RAM.

I would be happy to provide any additional info.
Comment 1 Ruediger Pluem 2014-01-08 15:58:41 UTC
Can you please check the patch found here:
http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_http.c?r1=1553540&r2=1553539&pathrev=1553540&view=patch
and let us know if it fixes your issue?
Comment 2 Ruediger Pluem 2014-01-08 15:59:17 UTC
In case they fix it you hit PR50335
Comment 3 Alex Frolkin 2014-01-08 16:40:54 UTC
I'll give it a go and report back tomorrow.  Thanks!
Comment 4 Alex Frolkin 2014-01-09 09:01:44 UTC
Yes, the patch does seem to have fixed the problem.  Thanks!

*** This bug has been marked as a duplicate of bug 50335 ***