Bug 63265 - does not check apr_bucket_read return value and then use uninitialized returned len value
Summary: does not check apr_bucket_read return value and then use uninitialized return...
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_deflate (show other bugs)
Version: 2.4.38
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: PatchAvailable
Depends on:
Blocks:
 
Reported: 2019-03-16 16:02 UTC by Sylvain Rochet
Modified: 2019-03-16 17:42 UTC (History)
1 user (show)



Attachments
patch adding checks to apr_bucket_read return value (2.47 KB, patch)
2019-03-16 16:02 UTC, Sylvain Rochet
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sylvain Rochet 2019-03-16 16:02:06 UTC
Hi,

For more time that I am willing to admit (years…) we had from time to time a child worker segfault, and each time I looked into it was always the same backtrace. We are using some of home made modules and I suspected the issue to be on my side because the backtrace sometimes showed calls to our modules. I prevented the issue to happen by returning a 0 length value from our modules when our module apr_bucket_read function does not succeed to mitigate the issue, considering at the time that the problem was on my side for not properly sanitizing the returned len variable, and it worked quite well.

But it did not completely fix the issue, but since the issue was not happening that often anymore I postponed again and again to look into it but recently for whatever reason we are hitting it more, really more.

So, the issue is mod_deflate does not check apr_bucket_read return value and then use the uninitialized len value. In this trace it is using uninitialized returned len value from mmap_bucket_read function.

# gdb /usr/sbin/apache2 core-apache2 
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug//usr/sbin/apache2...done.
done.
[New LWP 15175]
[New LWP 15180]
[New LWP 15179]
[New LWP 15182]
[New LWP 15181]
[New LWP 15178]
[New LWP 15188]
[New LWP 15183]
[New LWP 15193]
[New LWP 15194]
[New LWP 15187]
[New LWP 15192]
[New LWP 15184]
[New LWP 15189]
[New LWP 15198]
[New LWP 15190]
[New LWP 15177]
[New LWP 15199]
[New LWP 15200]
[New LWP 15201]
[New LWP 15202]
[New LWP 15197]
[New LWP 15185]
[New LWP 15196]
[New LWP 15186]
[New LWP 15195]
[New LWP 15191]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGBUS, Bus error.
#0  0xe3cf0440 in __kernel_vsyscall ()
(gdb) info threads
  Id   Target Id         Frame 
  27   Thread 0xdb04ab40 (LWP 15191) 0xe3cf0440 in __kernel_vsyscall ()
  26   Thread 0xd8d9ab40 (LWP 15195) 0xe3cf0440 in __kernel_vsyscall ()
  25   Thread 0xdda92b40 (LWP 15186) 0xe3cf0440 in __kernel_vsyscall ()
  24   Thread 0xd8581b40 (LWP 15196) 0xe3cf0440 in __kernel_vsyscall ()
  23   Thread 0xde2a9b40 (LWP 15185) 0xe3cf0440 in __kernel_vsyscall ()
  22   Thread 0xd7cfbb40 (LWP 15197) 0xe3cf0440 in __kernel_vsyscall ()
  21   Thread 0xd5397b40 (LWP 15202) 0xe3cf0440 in __kernel_vsyscall ()
  20   Thread 0xd5bc0b40 (LWP 15201) 0xe3cf0440 in __kernel_vsyscall ()
  19   Thread 0xd63cfb40 (LWP 15200) 0xe3cf0440 in __kernel_vsyscall ()
  18   Thread 0xd6cacb40 (LWP 15199) 0xe3cf0440 in __kernel_vsyscall ()
  17   Thread 0xe2726b40 (LWP 15177) 0xe3cf0440 in __kernel_vsyscall ()
  16   Thread 0xdb926b40 (LWP 15190) 0xe3cf0440 in __kernel_vsyscall ()
  15   Thread 0xd74e1b40 (LWP 15198) 0xe3cf0440 in __kernel_vsyscall ()
  14   Thread 0xdc1c4b40 (LWP 15189) 0xe3cf0440 in __kernel_vsyscall ()
  13   Thread 0xdead2b40 (LWP 15184) 0xe3cf0440 in __kernel_vsyscall ()
  12   Thread 0xda754b40 (LWP 15192) crc32_little (len=631606, buf=0xe2765000 <error: Cannot access memory at address 0xe2765000>, crc=0) at crc32.c:264
  11   Thread 0xdd237b40 (LWP 15187) 0xe3cf0440 in __kernel_vsyscall ()
  10   Thread 0xd968db40 (LWP 15194) 0xe3cf0440 in __kernel_vsyscall ()
  9    Thread 0xd9f0ab40 (LWP 15193) 0xe3cf0440 in __kernel_vsyscall ()
  8    Thread 0xdf360b40 (LWP 15183) 0xe3cf0440 in __kernel_vsyscall ()
  7    Thread 0xdc9dab40 (LWP 15188) 0xe3cf0440 in __kernel_vsyscall ()
  6    Thread 0xe1eb0b40 (LWP 15178) 0xe3cf0440 in __kernel_vsyscall ()
  5    Thread 0xe04cfb40 (LWP 15181) 0xe3cf0440 in __kernel_vsyscall ()
  4    Thread 0xdfc0db40 (LWP 15182) 0xe3cf0440 in __kernel_vsyscall ()
  3    Thread 0xe15c1b40 (LWP 15179) 0xe3cf0440 in __kernel_vsyscall ()
  2    Thread 0xe0d0fb40 (LWP 15180) 0xe3cf0440 in __kernel_vsyscall ()
* 1    Thread 0xe39d0740 (LWP 15175) 0xe3cf0440 in __kernel_vsyscall ()
(gdb) thread 12
[Switching to thread 12 (Thread 0xda754b40 (LWP 15192))]
#0  crc32_little (len=631606, buf=0xe2765000 <error: Cannot access memory at address 0xe2765000>, crc=0) at crc32.c:264
264	crc32.c: No such file or directory.
(gdb) bt full
#0  crc32_little (len=631606, buf=0xe2765000 <error: Cannot access memory at address 0xe2765000>, crc=0) at crc32.c:264
        c = 4294967295
        buf4 = 0xe2765004
#1  crc32 (crc=0, buf=<optimized out>, len=<optimized out>) at crc32.c:222
        endian = 1
#2  0xe394fbe2 in deflate_out_filter (f=0xe32b1c18, bb=0xe32b1eb0) at mod_deflate.c:943
        b = 0xe32af018
        e = 0xe36450e8
        r = 0xe32af058
        ctx = 0xe32b1f10
        zRC = 0
        len = 631606
        blen = 631606
        data = 0xe2765000 <error: Cannot access memory at address 0xe2765000>
        c = 0xe39447d8
#3  0xe393615a in filter_harness (f=0xe32b1c18, bb=0xe32b1eb0) at mod_filter.c:323
        ret = -483725288
        cachecontrol = 0xe3645018 "\030pd\343@\032\200", <incomplete sequence \342>
        ctx = 0xe32b1c30
        filter = 0xe357fa78
#4  0x0950cd5e in ap_pass_brigade (next=0xe32b1c18, bb=0xe32b1eb0) at util_filter.c:590
        e = 0xe3645140
#5  0x0951dc58 in default_handler (r=0xe32af058) at core.c:4513
        c = 0xe3647210
        bb = 0xe32b1eb0
        e = 0xe3645140
        d = 0xe32b0b78
        errstatus = 0
        fd = 0xe32b1d90
        status = 0
        bld_content_md5 = 0
#6  0x0952ad30 in ap_run_handler (r=0xe32af058) at config.c:169
        pHook = 0xe356b730
        n = 6
        rv = -1
#7  0x0952b6d8 in ap_invoke_handler (r=0xe32af058) at config.c:433
        handler = 0xe3576110 "application/xml"
        p = 0x0
        result = 0
        old_handler = 0x0
        ignore = 0xe32b0300 "\030\360*\343\001"
#8  0x0954517b in ap_process_async_request (r=0xe32af058) at http_request.c:317
        c = 0xe3647210
        access_status = 0
#9  0x09545262 in ap_process_request (r=0xe32af058) at http_request.c:363
        bb = 0xda7541c8
        b = 0x95712cc
        c = 0xe3647210
        rv = -479956464
#10 0x09541410 in ap_process_http_sync_connection (c=0xe3647210) at http_core.c:190
        r = 0xe32af058
        cs = 0x0
        csd = 0x0
        mpm_state = 0
#11 0x09541520 in ap_process_http_connection (c=0xe3647210) at http_core.c:231
No locals.
#12 0x095364e6 in ap_run_process_connection (c=0xe3647210) at connection.c:41
        pHook = 0xe356bad8
        n = 1
        rv = -1
#13 0x095369b0 in ap_process_connection (c=0xe3647210, csd=0xe3647060) at connection.c:203
        rc = -2
#14 0xe38f30e0 in process_socket (thd=0xe35c5588, p=0xe3647018, sock=0xe3647060, my_child_num=2, my_thread_num=15, bucket_alloc=0xe3645018) at worker.c:619
        current_conn = 0xe3647210
        conn_id = 143
        sbh = 0xe3647208
#15 0xe38f3e48 in worker_thread (thd=0xe35c5588, dummy=0xe2800510) at worker.c:978
        ti = 0xe2800510
        process_slot = 2
        thread_slot = 15
        csd = 0xe3647060
        bucket_alloc = 0xe3645018
        last_ptrans = 0x0
        ptrans = 0xe3647018
        rv = 0
        is_idle = 0
#16 0xe3c31c88 in dummy_worker (opaque=0xe35c5588) at /root/apr-1.5.1/threadproc/unix/thread.c:142
        thread = 0xe35c5588
#17 0xe3bf4ecb in start_thread (arg=0xda754b40) at pthread_create.c:309
        __res = <optimized out>
        pd = 0xda754b40
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-473927680, -629847232, 4001536, -629849112, -940106542, 1407995041}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#18 0xe3b2cd0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
No locals.
(gdb) f 2
#2  0xe394fbe2 in deflate_out_filter (f=0xe32b1c18, bb=0xe32b1eb0) at mod_deflate.c:943
943	mod_deflate.c: No such file or directory.
(gdb) print e->type->read
$1 = (apr_status_t (*)(apr_bucket *, const char **, apr_size_t *, apr_read_type_e)) 0xe3c49ffb <mmap_bucket_read>
(gdb)
Comment 1 Sylvain Rochet 2019-03-16 16:02:57 UTC
Created attachment 36489 [details]
patch adding checks to apr_bucket_read return value
Comment 2 Sylvain Rochet 2019-03-16 17:11:43 UTC
Well, it worked well for some time but I just hit it again with mmap_bucket_read returning SUCCESS (and a bad data pointer). There is something else triggering the issue. Dammit.

(gdb) print e->type->read
$5 = (apr_status_t (*)(apr_bucket *, const char **, apr_size_t *, apr_read_type_e)) 0xe9c24ffb <mmap_bucket_read>

(gdb) bt full
#0  crc32_little (len=682740, buf=0xd887a000 <error: Cannot access memory at address 0xd887a000>, crc=0) at crc32.c:264
        c = 4294967295
        buf4 = 0xd887a004
#1  crc32 (crc=0, buf=<optimized out>, len=<optimized out>) at crc32.c:222
        endian = 1
#2  0xe992bc68 in deflate_out_filter (f=0xd9b4ac18, bb=0xd9b4aeb0) at mod_deflate.c:948
        b = 0xe9914f6c
        e = 0xe4e060e8
        r = 0xe4da1058
        ctx = 0xd9b4af10
        zRC = 0
        len = 682740
        blen = 682740
        data = 0xd887a000 <error: Cannot access memory at address 0xd887a000>
        c = 0xe99207d8
        rv = 0
#3  0xe991215a in filter_harness (f=0xd9b4ac18, bb=0xd9b4aeb0) at mod_filter.c:323
        ret = -455471080
        cachecontrol = 0xe4e06018 "\030\200\340\344\340 \220", <incomplete sequence \350>
        ctx = 0xd9b4ac30
        filter = 0xe955ba78
#4  0x0a773d5e in ap_pass_brigade (next=0xd9b4ac18, bb=0xd9b4aeb0) at util_filter.c:590
        e = 0xe4e06140
#5  0x0a784c58 in default_handler (r=0xe4da1058) at core.c:4513
        c = 0xe4e08210
        bb = 0xd9b4aeb0
        e = 0xe4e06140
        d = 0xe4da2b78
        errstatus = 0
        fd = 0xd9b4ad90
        status = 0
        bld_content_md5 = 0
#6  0x0a791d30 in ap_run_handler (r=0xe4da1058) at config.c:169
        pHook = 0xe9547740
        n = 6
        rv = -1
#7  0x0a7926d8 in ap_invoke_handler (r=0xe4da1058) at config.c:433
        handler = 0xe9552130 "application/xml"
        p = 0x0
        result = 0
        old_handler = 0x0
        ignore = 0xe4da2300 "\030\020\332\344\001"
#8  0x0a7ac17b in ap_process_async_request (r=0xe4da1058) at http_request.c:317
        c = 0xe4e08210
        access_status = 0
#9  0x0a7ac262 in ap_process_request (r=0xe4da1058) at http_request.c:363
        bb = 0xe3c591c8
        b = 0xa7d82cc
        c = 0xe4e08210
        rv = -455048688
#10 0x0a7a8410 in ap_process_http_sync_connection (c=0xe4e08210) at http_core.c:190
        r = 0xe4da1058
        cs = 0x0
        csd = 0x0
        mpm_state = 0
#11 0x0a7a8520 in ap_process_http_connection (c=0xe4e08210) at http_core.c:231
No locals.
#12 0x0a79d4e6 in ap_run_process_connection (c=0xe4e08210) at connection.c:41
        pHook = 0xe9547ae8
        n = 1
        rv = -1
#13 0x0a79d9b0 in ap_process_connection (c=0xe4e08210, csd=0xe4e08060) at connection.c:203
        rc = -2
#14 0xe98cf0e0 in process_socket (thd=0xe95a14c8, p=0xe4e08018, sock=0xe4e08060, my_child_num=1, my_thread_num=9, bucket_alloc=0xe4e06018) at worker.c:619
        current_conn = 0xe4e08210
        conn_id = 73
        sbh = 0xe4e08208
#15 0xe98cfe48 in worker_thread (thd=0xe95a14c8, dummy=0xe8900a30) at worker.c:978
        ti = 0xe8900a30
        process_slot = 1
        thread_slot = 9
        csd = 0xe4e08060
        bucket_alloc = 0xe4e06018
        last_ptrans = 0x0
        ptrans = 0xe4e08018
        rv = 0
        is_idle = 0
#16 0xe9c0cc88 in dummy_worker (opaque=0xe95a14c8) at /root/apr-1.5.1/threadproc/unix/thread.c:142
        thread = 0xe95a14c8
#17 0xe9bcfecb in start_thread (arg=0xe3c59b40) at pthread_create.c:309
        __res = <optimized out>
        pd = 0xe3c59b40
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-373415936, -473588928, 4001536, -473590808, 640027362, -721593610}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#18 0xe9b07d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
No locals.
(gdb)