Bug 59749 - apache http 2.2.29: Segmentation fault
Summary: apache http 2.2.29: Segmentation fault
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.2.29
Hardware: All Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate
Depends on:
Blocks:
 
Reported: 2016-06-23 22:30 UTC by Rahul
Modified: 2018-11-07 21:08 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rahul 2016-06-23 22:30:28 UTC
We are using apache http server 2.2.29 on Linux (2.6.32).We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:

[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name
[Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name

 Below is the output of the httpd -V

./httpd -V
 Server version: Apache/2.2.29 (Unix)
 Server built:   Jan 13 2016 15:10:57
 Server's Module Magic Number: 20051115:36
 Server loaded:  APR 1.5.1, APR-Util 1.5.3
 Compiled using: APR 1.5.1, APR-Util 1.5.3
 Architecture:   64-bit
 Server MPM:     Worker
 threaded:     yes (fixed thread count)
 forked:     yes (variable process count)
 Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/app/****"
 -D SUEXEC_BIN="/app/****/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"


Below is the output of 'bt full' [ during the core analysis ]

#0  0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
     No symbol table info available.
    #1  0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>)     at pod.c:54
    c = <value optimized out>
    fd = 6
    rc = <value optimized out>
   #2  0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
    threads = 0xc2bab0
    rv = <value optimized out>
    ts = 0xc079f8
    thread_attr = 0xc07a18
    start_thread_id = <value optimized out>
  #3  0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
    pid = 0
   #4  0x000000000044efc0 in perform_idle_server_maintenance     (_pconf=<value             optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
    j = <value optimized out>
    free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137,   11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
      4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
    total_non_dead = <value optimized out>
    active_thread_count = <value optimized out>
    idle_thread_count = <value optimized out>
    free_length = 1
    i = <value optimized out>
    ps = <value optimized out>
    totally_free_length = <value optimized out>
    last_non_dead = <value optimized out>
      #5  server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
    status = 11
    pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
    i = <value optimized out>
    child_slot = <value optimized out>
    exitwhy = 6
    processed_status = <value optimized out>
    #6  ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>,     s=<value optimized out>) at worker.c:1782
    remaining_children_to_start = <value optimized out>
    rv = <value optimized out>
    #7  0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
    c = 0 '\000'
    configtestonly = <value optimized out>
    confname = 0x458a64 "conf/httpd.conf"
    def_server_root = 0x458a4f "/app/***"
    temp_error_log = 0x0
    error = <value optimized out>
    process = 0xb44870
    server_conf = 0xb44870
    pglobal = 0xb3b128
    pconf = 0xb3d138
    plog = 0xb45178
    ptemp = 0xb41158
    pcommands = 0xb3f148
    opt = 0xb3f238
    rv = <value optimized out>
    mod = <value optimized out>
    optarg = 0x0
    signal_server = <value optimized out>'

We are using Mod_proxy as a balancer to the backend jboss application server.

I have attached a core dump herewith.
Comment 1 Rahul 2016-06-23 22:34:38 UTC
Unable to add core dump due to size limit
Comment 2 Eric Covener 2016-06-23 22:43:10 UTC
> Below is the output of 'bt full' [ during the core analysis ]
> 
> #0  0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
>      No symbol table info available.
>     #1  0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>)  
> at pod.c:54
>     c = <value optimized out>
>     fd = 6
>     rc = <value optimized out>
>    #2  0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
>     threads = 0xc2bab0

wrong thread, there's probably a more interesting one.
Comment 3 Rahul 2016-06-24 13:40:18 UTC
Below is the 'bt full' output of another core? Can this help?

#0  0x0000003037f2b4d5 in __strcasecmp_l_sse42 () from /lib64/libc.so.6
No symbol table info available.
#1  0x00002b9f1c7c67b4 in ap_proxy_determine_connection (p=0x2b9f3001d0a8, r=0x2b9f3001d120, conf=0x80fdb8, worker=0x7e13c8, conn=0x832ff8,
    uri=<value optimized out>, url=0x2b9f213c3b48, proxyname=0x0, proxyport=0, server_portstr=0x2b9f213c3b50 "", server_portstr_size=32)
    at proxy_util.c:2228
        ssl_hostname = 0x2b9f3403cf00 "rchvasp0037.rsrpmi.net"
        server_port = <value optimized out>
        err = <value optimized out>
        uerr = <value optimized out>
#2  0x00002b9f1cddb26d in proxy_http_handler (r=0x2b9f3001d120, worker=0x7e13c8, conf=0x80fdb8, url=0x2b9f30020158 "/CommonWeb/pages/common/jsp/login.jsp",
    proxyname=0x0, proxyport=0) at mod_proxy_http.c:2024
        status = 0
        server_portstr = "\000\321\001\060\237+\000\000\370\376\200", '\000' <repeats 13 times>"\370, \376\200\000\000\000\000"
        scheme = <value optimized out>
        proxy_function = 0x2b9f1cddd6ac "HTTPS"
        u = <value optimized out>
        backend = 0x832ff8
        is_ssl = <value optimized out>
        c = 0x2b9f2c000c48
        p = 0x2b9f3001d0a8
        uri = 0x2b9f30020068
#3  0x00002b9f1c7bde62 in proxy_run_scheme_handler (r=0x2b9f3001d120, worker=0x7e13c8, conf=0x80fdb8,
    url=0x2b9f30020018 "https://rchvasp0037.rsrpmi.net:8443/CommonWeb/pages/common/jsp/login.jsp", proxyhost=0x0, proxyport=0) at mod_proxy.c:2412
        pHook = <value optimized out>
        n = <value optimized out>
        rv = -1
#4  0x00002b9f1c7c2417 in proxy_handler (r=0x2b9f3001d120) at mod_proxy.c:1039
        url = 0x2b9f30020018 "https://rchvasp0037.rsrpmi.net:8443/CommonWeb/pages/common/jsp/login.jsp"
        uri = 0x2b9f3003bede "balancer://paascluster/CommonWeb/pages/common/jsp/login.jsp"
        scheme = 0x2b9f3003bf78 "balancer"
        p = 0x2b9f213c3d08 "\250\323\001\060\237+"
        p2 = <value optimized out>
        sconf = <value optimized out>
        conf = 0x80fdb8
        proxies = 0x80fef8
        ents = 0x80ff18
        i = <value optimized out>
        access_status = 0
        direct_connect = <value optimized out>
        str = <value optimized out>
        maxfwd = <value optimized out>
        balancer = 0x80cf78
        worker = 0x7e13c8
        attempts = 0
        max_attempts = 1
        list = <value optimized out>
#5  0x00000000004399f0 in ap_run_handler (r=0x2b9f3001d120) at config.c:158
        pHook = <value optimized out>
        n = <value optimized out>
        rv = -1
#6  0x000000000043d01e in ap_invoke_handler (r=0x2b9f3001d120) at config.c:376
        handler = <value optimized out>
        p = <value optimized out>
        result = 0
        old_handler = 0x2b9f2055a457 "proxy-server"
        ignore = <value optimized out>
#7  0x0000000000447d60 in ap_process_request (r=0x2b9f3001d120) at http_request.c:282
        access_status = <value optimized out>
#8  0x0000000000444ce0 in ap_process_http_connection (c=0x2b9f2c000c48) at http_core.c:190
Comment 4 Yann Ylavic 2016-06-27 06:21:49 UTC
It seems that the proxy_conn_rec is corrupted (at least conn->ssl_hostname), but I don't see how this can happen.

Could you please "print *conn" in ap_proxy_determine_connection (frame 1) and show the output?
Comment 5 Rahul 2016-06-27 11:46:13 UTC
Could you please provide me the steps to perform this please?
Comment 6 Rahul 2016-06-27 11:46:48 UTC
(In reply to Yann Ylavic from comment #4)
> It seems that the proxy_conn_rec is corrupted (at least conn->ssl_hostname),
> but I don't see how this can happen.
> 
> Could you please "print *conn" in ap_proxy_determine_connection (frame 1)
> and show the output?

Could you please provide me the steps to perform this please?
Comment 7 Yann Ylavic 2016-06-27 12:40:22 UTC
(In reply to Rahul from comment #6)
> (In reply to Yann Ylavic from comment #4)
> > 
> > Could you please "print *conn" in ap_proxy_determine_connection (frame 1)
> > and show the output?
> 
> Could you please provide me the steps to perform this please?

In gdb, according to comment 3, you should be able to switch to function ap_proxy_determine_connection by typing (where "(gdb)" is the prompt):
(gdb) frame 1

and then simply enter:
(gdb) print *conn
Comment 8 Rahul 2016-06-27 13:22:47 UTC
(In reply to Yann Ylavic from comment #7)
> (In reply to Rahul from comment #6)
> > (In reply to Yann Ylavic from comment #4)
> > > 
> > > Could you please "print *conn" in ap_proxy_determine_connection (frame 1)
> > > and show the output?
> > 
> > Could you please provide me the steps to perform this please?
> 
> In gdb, according to comment 3, you should be able to switch to function
> ap_proxy_determine_connection by typing (where "(gdb)" is the prompt):
> (gdb) frame 1
> 
> and then simply enter:
> (gdb) print *conn

I have have taken the 'bt full' of a fresh core dump. The "print *conn" output for this core dump have also followed below.

Please note that "xxxxx.xxxx.net" is hostname of the bakcend jboss application server .

(gdb) bt full
#0  ap_pass_brigade (next=0x200000000, bb=0x2b9f3803e5c0) at util_filter.c:526
        e = <value optimized out>
#1  0x00002b9f1cdd99ca in pass_brigade (bucket_alloc=<value optimized out>, r=0x2b9f4402e220, conn=0x2b9f24027bb8, origin=0x2b9f44001e20,
    bb=0x2b9f3803e5c0, flush=<value optimized out>) at mod_proxy_http.c:269
        status = <value optimized out>
        transferred = 1638
#2  0x00002b9f1cddc017 in stream_reqbody_cl (r=0x2b9f4402e220, worker=<value optimized out>, conf=0x2b9f231d2bd8,
    url=0x2b9f3803e4e0 "/LoginWeb/ui/jsp/headerMenu.xhtml?isDbAbsent=null&siteName=null&homePage=Yes&type=login&userName=Dave%20Baker&lastLoginDetails=First%20Time%20LogIn&envName=PRDBANCS2&version=Client_12.01.00.01.169&onsit"..., proxyname=0x2b9f3802d718 "Xs\001,\237+", proxyport=14016) at mod_proxy_http.c:537
        rv = <value optimized out>
        status = <value optimized out>
        bucket_alloc = 0x2b9f3802d718
        bb = 0x2b9f3803e5c0
        e = <value optimized out>
        bytes = 47963040834080
        seen_eos = <value optimized out>
        cl_val = 0
        bytes_streamed = 0
#3  ap_proxy_http_request (r=0x2b9f4402e220, worker=<value optimized out>, conf=0x2b9f231d2bd8,
    url=0x2b9f3803e4e0 "/LoginWeb/ui/jsp/headerMenu.xhtml?isDbAbsent=null&siteName=null&homePage=Yes&type=login&userName=Dave%20Baker&lastLoginDetails=First%20Time%20LogIn&envName=PRDBANCS2&version=Client_12.01.00.01.169&onsit"..., proxyname=0x2b9f3802d718 "Xs\001,\237+", proxyport=14016) at mod_proxy_http.c:1143
        bucket_alloc = 0x2b9f231d2bb8
        e = <value optimized out>
        headers_in_array = <value optimized out>
        rv = <value optimized out>
        temp_brigade = 0x2b9f4402a768
        force10 = 589114312
        headers_in_copy = <value optimized out>
        c = 0x2b9f2c0175f8
        header_brigade = 0x2b9f3803e5c0
        input_brigade = 0x2b9f4402a728
        buf = <value optimized out>
        headers_in = 0x6736c0
        status = <value optimized out>
        bytes_read = <value optimized out>
        bytes = 0
        counter = 589114304
        rb_method = <value optimized out>
        old_cl_val = <value optimized out>
        old_te_val = 0x2b9f3802d718 "Xs\001,\237+"
#4  proxy_http_handler (r=0x2b9f4402e220, worker=<value optimized out>, conf=0x2b9f231d2bd8,
    url=0x2b9f3803e4e0 "/LoginWeb/ui/jsp/headerMenu.xhtml?isDbAbsent=null&siteName=null&homePage=Yes&type=login&userName=Dave%20Baker&lastLoginDetails=First%20Time%20LogIn&envName=PRDBANCS2&version=Client_12.01.00.01.169&onsit"..., proxyname=0x2b9f3802d718 "Xs\001,\237+", proxyport=14016) at mod_proxy_http.c:2054
        status = <value optimized out>
        server_portstr = "\000\342\002D\237+\000\000\370\376\200", '\000' <repeats 13 times>"\370, \376\200\000\000\000\000"
        scheme = <value optimized out>
        proxy_function = 0x2b9f1cddd6ac "HTTPS"
        u = <value optimized out>
        backend = 0x2b9f24027bb8
        is_ssl = <value optimized out>
        c = 0x2b9f2c0175f8
        p = 0x2b9f4402e1a8
        uri = <value optimized out>
#5  0x00002b9f1c7bde62 in proxy_run_scheme_handler (r=0x2b9f4402e220, worker=0x7e14c8, conf=0x80fdb8,
    url=0x2b9f3803e228 "https://xxxxx.xxxx.net:8443/LoginWeb/ui/jsp/headerMenu.xhtml?isDbAbsent=null&siteName=null&homePage=Yes&type=login&userName=Dave%20Baker&lastLoginDetails=First%20Time%20LogIn&envName=PRDBANCS2"..., proxyhost=0x0, proxyport=0) at mod_proxy.c:2412
        pHook = <value optimized out>
       n = <value optimized out>
        rv = -1


(gdb) frame 1
#1  0x00002b9f1cdd99ca in pass_brigade (bucket_alloc=<value optimized out>, r=0x2b9f4402e220, conn=0x2b9f24027bb8, origin=0x2b9f44001e20,
    bb=0x2b9f3803e5c0, flush=<value optimized out>) at mod_proxy_http.c:269
269         status = ap_pass_brigade(origin->output_filters, bb);
(gdb) print *conn
$1 = {connection = 0x2b9f44001e20, hostname = 0x2b9f3400d9a0 "xxxxx.xxxx.net", port = 8443, is_ssl = 1, pool = 0x2b9f3400d928,
  sock = 0x2b9f2403f2a8, addr = 0x2b9f240273c8, flags = 0, close = 0, close_on_recycle = 0, worker = 0x7e14c8, data = 0x0, inreslist = 0,
  scpool = 0x2b9f2403f218, r = 0x0, need_flush = 0, forward = 0x0, ssl_hostname = 0x2b9f2403f290 "xxxxx.xxxx.net"}
Comment 9 Rainer Jung 2018-02-25 19:51:11 UTC
Undo spam change
Comment 10 William A. Rowe Jr. 2018-11-07 21:08:03 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.