Bug 54785

Summary: Random crash when httpd server is stopped
Product: Apache httpd-2 Reporter: gabriel
Component: CoreAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: NEW ---    
Severity: normal    
Priority: P2    
Version: 2.4.4   
Target Milestone: ---   
Hardware: Other   
OS: Linux   
Attachments: error_log

Description gabriel 2013-04-02 12:48:51 UTC
Almost every time I stop httpd server I get this kind of errors:

# cat /var/log/httpd/error_log
[Tue Apr 02 08:33:34.714313 2013] [mpm_prefork:notice] [pid 6720] AH00170: caught SIGWINCH, shutting down gracefully
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6678624798 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6678624798 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6678624798 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f6678624798 ***
======= Backtrace: =========
======= Backtrace: =========
/lib64/libc.so.6(+0x7ca8e)[0x7f66782efa8e]
======= Backtrace: =========
======= Backtrace: =========
/lib64/libapr-1.so.0(apr_pool_destroy+0x19c)[0x7f6678a6485c]
/etc/httpd/modules/mod_mpm_prefork.so(+0x31be)[0x7f667312c1be]
/lib64/libc.so.6(+0x7ca8e)[0x7f66782efa8e]
/etc/httpd/modules/mod_mpm_prefork.so(+0x39cb)[0x7f667312c9cb]
/lib64/libpthread.so.0(+0xf000)[0x7f667883e000]
/lib64/libapr-1.so.0(apr_pool_destroy+0x19c)[0x7f6678a6485c]
/lib64/ld-linux-x86-64.so.2(+0xf5f7)[0x7f66799215f7]
/etc/httpd/modules/mod_mpm_prefork.so(+0x31be)[0x7f667312c1be]
/lib64/libc.so.6(+0x38df1)[0x7f66782abdf1]
/etc/httpd/modules/mod_mpm_prefork.so(+0x39cb)[0x7f667312c9cb]
/lib64/libc.so.6(+0x38e75)[0x7f66782abe75]
/lib64/libpthread.so.0(+0xf000)[0x7f667883e000]
/etc/httpd/modules/mod_version.so(+0xa80)[0x7f6673534a80]
======= Memory map: ========
/etc/httpd/modules/mod_mpm_prefork.so(+0x31db)[0x7f667312c1db]
/etc/httpd/modules/mod_mpm_prefork.so(+0x35f5)[0x7f667312c5f5]
/etc/httpd/modules/mod_mpm_prefork.so(+0x394c)[0x7f667312c94c]
/etc/httpd/modules/mod_mpm_prefork.so(+0x39a6)[0x7f667312c9a6]
/etc/httpd/modules/mod_mpm_prefork.so(+0x4856)[0x7f667312d856]
/usr/sbin/httpd(ap_run_mpm+0x4e)[0x7f6679b6e6de]
....
(see attached error_log file).

# find /tmp -name core* 
/tmp/systemd-private-hjmsNN/core.6725
/tmp/systemd-private-hjmsNN/core.6724
/tmp/systemd-private-hjmsNN/core.6726
/tmp/systemd-private-hjmsNN/core.6722

# gdb httpd -c /tmp/systemd-private-hjmsNN/core.6722
GNU gdb (GDB) Fedora (7.5.1-37.fc18)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done.
done.
[New LWP 6722]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 6, Aborted.
#0  0x00007f66782a8ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63
63        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) where
#0  0x00007f66782a8ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63
#1  0x00007f66782aa358 in __GI_abort () at abort.c:90
#2  0x00007f66782e859b in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x7f66783ecba8 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:197
#3  0x00007f66782efa8e in malloc_printerr (ptr=0x7f6678624798 <main_arena+88>, 
    str=0x7f66783eaa01 "free(): invalid pointer", action=3) at malloc.c:4969
#4  _int_free (av=0x7f6678624740 <main_arena>, p=0x7f6678624788 <main_arena+72>, have_lock=0) at malloc.c:3826
#5  0x00007f6678a6485c in allocator_free (node=0x7f6678624798 <main_arena+88>, allocator=0x7f667bf3b7d0)
    at memory/unix/apr_pools.c:430
#6  apr_pool_destroy (pool=0x7f667bf3b8c8) at memory/unix/apr_pools.c:856
#7  0x00007f667312c1be in clean_child_exit (code=code@entry=0) at prefork.c:218
#8  0x00007f667312c9cb in just_die (sig=<optimized out>) at prefork.c:344
#9  <signal handler called>
#10 0x00007f6673534a80 in __do_global_dtors_aux () from /etc/httpd/modules/mod_version.so
#11 0x00007f66799215f7 in _dl_fini () at dl-fini.c:253
#12 0x00007f66782abdf1 in __run_exit_handlers (status=status@entry=0, listp=0x7f66786246a8 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#13 0x00007f66782abe75 in __GI_exit (status=status@entry=0) at exit.c:99
#14 0x00007f667312c1db in clean_child_exit (code=code@entry=0) at prefork.c:227
#15 0x00007f667312c7c5 in child_main (child_num_arg=child_num_arg@entry=0) at prefork.c:629
#16 0x00007f667312c94c in make_child (s=0x7f667bda0348, slot=slot@entry=0) at prefork.c:800
#17 0x00007f667312c9a6 in startup_children (number_to_start=5) at prefork.c:818
#18 0x00007f667312d856 in prefork_run (_pconf=<optimized out>, plog=0x7f667bdd24d8, s=0x7f667bda0348) at prefork.c:976
#19 0x00007f6679b6e6de in ap_run_mpm (pconf=0x7f667bd75138, plog=0x7f667bdd24d8, s=0x7f667bda0348) at mpm_common.c:98
#20 0x00007f6679b67d8a in main (argc=1, argv=0x7fff2a861578) at main.c:777
(gdb) bt full
#0  0x00007f66782a8ba5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:63
        resultvar = 0
        pid = 6722
        selftid = 6722
#1  0x00007f66782aa358 in __GI_abort () at abort.c:90
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x7f66783eabc1, sa_sigaction = 0x7f66783eabc1}, sa_mask = {__val = {
              3, 140733906812557, 3, 140078080761236, 1, 140078080767934, 3, 140733906812532, 12, 140078080767938, 2, 
              140078080767938, 2, 140733906813344, 7, 140733906815104}}, sa_flags = 88, sa_restorer = 0x7}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f66782e859b in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x7f66783ecba8 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:197
        ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fff2a860890, reg_save_area = 0x7fff2a8607a0}}
        ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7fff2a860890, 
            reg_save_area = 0x7fff2a8607a0}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x00007f66782efa8e in malloc_printerr (ptr=0x7f6678624798 <main_arena+88>, 
    str=0x7f66783eaa01 "free(): invalid pointer", action=3) at malloc.c:4969
        buf = "00007f6678624798"
        cp = <optimized out>
#4  _int_free (av=0x7f6678624740 <main_arena>, p=0x7f6678624788 <main_arena+72>, have_lock=0) at malloc.c:3826
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = 0x7f66783eaa01 "free(): invalid pointer"
        locked = <optimized out>
#5  0x00007f6678a6485c in allocator_free (node=0x7f6678624798 <main_arena+88>, allocator=0x7f667bf3b7d0)
    at memory/unix/apr_pools.c:430
        freelist = 0x0
        max_index = <optimized out>
        max_free_index = <optimized out>
        next = <optimized out>
        index = <optimized out>
        current_free_index = <optimized out>
#6  apr_pool_destroy (pool=0x7f667bf3b8c8) at memory/unix/apr_pools.c:856
        active = <optimized out>
        allocator = 0x7f667bf3b7d0
#7  0x00007f667312c1be in clean_child_exit (code=code@entry=0) at prefork.c:218
No locals.
#8  0x00007f667312c9cb in just_die (sig=<optimized out>) at prefork.c:344
No locals.
#9  <signal handler called>
No locals.
#10 0x00007f6673534a80 in __do_global_dtors_aux () from /etc/httpd/modules/mod_version.so
No symbol table info available.
#11 0x00007f66799215f7 in _dl_fini () at dl-fini.c:253
        array = 0x7f6673735cb8
        i = 0
        nmaps = 65
        nloaded = <optimized out>
        i = 44
        l = 0x7f667bde12b0
        ns = 0
        maps = 0x7fff2a860de0
        maps_size = 520
        do_audit = 0
#12 0x00007f66782abdf1 in __run_exit_handlers (status=status@entry=0, listp=0x7f66786246a8 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true) at exit.c:77
        atfct = <optimized out>
        onfct = <optimized out>
        cxafct = <optimized out>
        f = <optimized out>
#13 0x00007f66782abe75 in __GI_exit (status=status@entry=0) at exit.c:99
No locals.
#14 0x00007f667312c1db in clean_child_exit (code=code@entry=0) at prefork.c:227
No locals.
#15 0x00007f667312c7c5 in child_main (child_num_arg=child_num_arg@entry=0) at prefork.c:629
        numdesc = 1
        pdesc = 0x7f667bf3d950
        current_conn = <optimized out>
        csd = 0x7f667bf3d950
        thd = 0x7f667bf3b940
        osthd = 140078105122688
        ptrans = 0x7f667bf3d8d8
        allocator = 0x7f667bf3b7d0
        status = <optimized out>
        i = <optimized out>
        lr = <optimized out>
        pollset = 0x7f667bf3ba00
        sbh = 0x7f667bf3b9f8
        bucket_alloc = 0x7f667bf418f8
        last_poll_idx = 1
        lockfile = <optimized out>
#16 0x00007f667312c94c in make_child (s=0x7f667bda0348, slot=slot@entry=0) at prefork.c:800
        pid = 0
#17 0x00007f667312c9a6 in startup_children (number_to_start=5) at prefork.c:818
        i = 0
#18 0x00007f667312d856 in prefork_run (_pconf=<optimized out>, plog=0x7f667bdd24d8, s=0x7f667bda0348) at prefork.c:976
        index = <optimized out>
        remaining_children_to_start = <optimized out>
        rv = <optimized out>
#19 0x00007f6679b6e6de in ap_run_mpm (pconf=0x7f667bd75138, plog=0x7f667bdd24d8, s=0x7f667bda0348) at mpm_common.c:98
        pHook = 0x7f667be4a618
        n = 0
        rv = 1936940472
#20 0x00007f6679b67d8a in main (argc=1, argv=0x7fff2a861578) at main.c:777
        c = 0 '\000'
        showcompile = 0
        showdirectives = 0
        confname = 0x7f6679ba2baf "conf/httpd.conf"
        def_server_root = 0x7f6679ba2ba4 "/etc/httpd"
        temp_error_log = <optimized out>
        error = <optimized out>
        process = 0x7f667bd73218
        pconf = 0x7f667bd75138
        plog = 0x7f667bdd24d8
        ptemp = 0x7f667bd9e2f8
        pcommands = 0x7f667bd97248
        opt = 0x7f667bd97338
        rv = <optimized out>
        mod = 0x7f6679dbf098 <ap_prelinked_modules+24>
        opt_arg = 0x7f667bd73128 "\b\361\326{f\177"
        signal_server = <optimized out>

Running Fedora 18.

# uname -a
Linux f18.vlasiu.net 3.8.1-201.fc18.x86_64 #1 SMP Thu Feb 28 19:23:08 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Does not matter if I use the httpd package from distribution or I compile httpd from sources. Same behaviour.
Also, module mod_systemd.so is not loaded!
Comment 1 gabriel 2013-04-02 12:49:38 UTC
Created attachment 30134 [details]
error_log