Bug 30134 - Segmentation fault in char_buffer_read when reverse proxying SSL
Segmentation fault in char_buffer_read when reverse proxying SSL
Status: CLOSED FIXED
Product: Apache httpd-2
Classification: Unclassified
Component: mod_ssl
2.0.50
Sun Solaris
: P3 major with 6 votes (vote)
: ---
Assigned To: Apache HTTPD Bugs Mailing List
http://www.2dv.cl/cepechphp/admin/ind...
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2004-07-15 23:57 UTC by M. "Alex" Hankins
Modified: 2005-03-15 09:46 UTC (History)
0 users



Attachments
proposed fix (924 bytes, patch)
2004-08-17 12:48 UTC, Joe Orton
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description M. "Alex" Hankins 2004-07-15 23:57:41 UTC
Overview Description:

Intermittent segmentation faults occur in char_buffer_read at
ssl_engine_io.c:348 when using a RewriteRule to do reverse proxying to an SSL
origin server running IIS.

    Steps to Reproduce:

1) Set up an IIS server using SSL and running eRoom 6.

2) Add the following directives to httpd.conf:

Listen 47290
SSLProxyEngine on
RewriteEngine on
RewriteRule /(.*) https://some.eroom6.iis.server.com/$1 [P]

3) Visit a URL similar to the following:

http://reverse.proxy.com:47290/eRoomASP/CookieTest.asp?facility=facility&URL=%2FeRoom%2FFacility%2FRoom%2F0_4242

If that doesn't cause the segfault, click around for a while.

(Yes, reverse proxying from non-SSL to SSL is not a good idea, but it keeps the
example simpler.)

    Actual Results:

Segmentation fault in error_log:
[Thu Jul 15 19:38:36 2004] [notice] child pid 42 exit signal Segmentation fault
(11), possible coredump in /usr/local/httpd-2.0.50

    Build Date & Platform:

2004-07-14 build on SunOS 5.8 SUNW,UltraAX-i2

    Additional Information:

Here is a stack trace from gdb:

#0  0xfef5060c in memcpy ()
   from /usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1
#1  0xfeafef54 in char_buffer_read (buffer=0x1649ac,
    in=0x2000 <Address 0x2000 out of bounds>, inl=8192) at ssl_engine_io.c:348
#2  0xfeaff388 in ssl_io_input_read (inctx=0x164990,
    buf=0x1649b8 "Content-Length: 121\r\nCort/Martonia/0_2615\r\nContent-Length:
121\r\nCent-Length: 121\r\nCmeport/Martonia/0_2615\r\nContent-Length:
121\r\nCmeport/Martonia/0_2615\r\nContent-Length:
121\r\nCrtonia/0_2615\r\nContent-Le"..., len=0xffbea8cc) at ssl_engine_io.c:561
#3  0xfeaff624 in ssl_io_input_getline (inctx=0x164990,
    buf=0x1649b8 "Content-Length: 121\r\nCort/Martonia/0_2615\r\nContent-Length:
121\r\nCent-Length: 121\r\nCmeport/Martonia/0_2615\r\nContent-Length:
121\r\nCmeport/Martonia/0_2615\r\nContent-Length:
121\r\nCrtonia/0_2615\r\nContent-Le"..., len=0xffbea944) at ssl_engine_io.c:712
#4  0xfeb00118 in ssl_io_filter_input (f=0x1669c0, bb=0x158f98,
    mode=4290685252, block=APR_BLOCK_READ, readbytes=0) at ssl_engine_io.c:1226
#5  0x42978 in ap_get_brigade (next=0x1669c0, bb=0x158f98,
    mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
    at util_filter.c:474
#6  0x4aab4 in net_time_filter (f=0x158e20, b=0x158f98, mode=AP_MODE_GETLINE,
    block=APR_BLOCK_READ, readbytes=0) at core.c:3600
#7  0x42978 in ap_get_brigade (next=0x158e20, bb=0x158f98,
    mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
    at util_filter.c:474
#8  0x43e0c in ap_rgetline_core (s=0xffbeab94, n=8192, read=0xffbeab90,
    r=0x1671b8, fold=1, bb=0x158f98) at protocol.c:214
#9  0x441a4 in ap_getline (s=0xffbeccd8 "Content-Length", n=8192, r=0x1671b8,
    fold=1) at protocol.c:478
#10 0xfe8552d4 in ap_proxy_read_headers (r=0x1821d0, rr=0x1671b8,
    buffer=0xffbeccd8 "Content-Length", size=8192, c=0x1671b8)
    at proxy_util.c:457
#11 0xfe833014 in ap_proxy_http_process_response (p=0x157960, r=0x1821d0,
    p_conn=0x157ee8, origin=0x158168, backend=0x157f00, conf=0xf0268,
    bb=0x157e98, server_portstr=0xffbeed68 ":47290") at proxy_http.c:755
#12 0xfe833ba4 in ap_proxy_http_handler (r=0x1821d0, conf=0xf0268,
    url=0x158038
"/eRoomASP/CookieTest.asp?facility=memeport&URL=%2FeRoom%2Fmemeport%2FMartonia%2F0_2615",
proxyname=0x0, proxyport=60776) at proxy_http.c:1121
#13 0xfe85435c in proxy_run_scheme_handler (r=0x1821d0, conf=0xf0268,
    url=0x1839ce
"https://eroomhost.aaa.bbb.com/eRoomASP/CookieTest.asp?facility=memeport&URL=%2FeRoom%2Fmemeport%2FMartonia%2F0_2615",
proxyhost=0x0,
    proxyport=0) at mod_proxy.c:1113
#14 0xfe852ed8 in proxy_handler (r=0x1821d0) at mod_proxy.c:418
#15 0x359a8 in ap_run_handler (r=0x1821d0) at config.c:151
#16 0x35fa4 in ap_invoke_handler (r=0x1821d0) at config.c:358
#17 0x32c44 in ap_process_request (r=0x1821d0) at http_request.c:246
#18 0x2df14 in ap_process_http_connection (c=0x157a70) at http_core.c:250
#19 0x40090 in ap_run_process_connection (c=0x157a70) at connection.c:42
#20 0x403a4 in ap_process_connection (c=0x157a70, csd=0x157998)
    at connection.c:175
#21 0x3422c in child_main (child_num_arg=5) at prefork.c:609
#22 0x343ac in make_child (s=0x8edb0, slot=5) at prefork.c:703
#23 0x345fc in perform_idle_server_maintenance (p=0x8c690) at prefork.c:838
#24 0x34a34 in ap_mpm_run (_pconf=0x0, plog=0x63400, s=0x83000)
    at prefork.c:1039
#25 0x3ad44 in main (argc=3, argv=0xffbef4ac) at main.c:617
Comment 1 Nick Kew 2004-07-16 05:12:34 UTC
Does this still happen if you take mod_rewrite out of the setup and use
ProxyPass instead?
Comment 2 M. "Alex" Hankins 2004-07-20 18:11:48 UTC
Yes, I still get what seems to be the same segfault (according to gdb) if I
replace these config lines:

RewriteEngine on
RewriteRule /(.*) https://some.eroom6.iis.server.com/$1 [P]

with this one:

ProxyPass / https://some.eroom6.iis.server.com/
Comment 3 Joe Orton 2004-08-11 13:30:29 UTC
This backtrace is a little strange:

#1  0xfeafef54 in char_buffer_read (buffer=0x1649ac,
    in=0x2000 <Address 0x2000 out of bounds>, inl=8192) at ssl_engine_io.c:348

which means either stack corruption by memcpy or in got passed in as a pointer
to address 8192 somehow.

Can you, from gdb against a core dump, check:

up 2 (into ssl_io_input_read)
print *inctx
info locals
print buf
print len


Comment 4 Stephan Tesch 2004-08-16 11:54:28 UTC
Hi Joe, 
 
I do suffer from the same bug. Attached is the info you requested. If you need 
additional testing, please say so. I cleared some info from the contents of 
buffer to not disclose critical information. I hope that's ok and doesn't 
hinder your debugging process. 
 
Regards, Stephan 
 
Program received signal SIGSEGV, Segmentation fault. 
0xfedf060c in memcpy () from /usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1 
(gdb) where full 
#0  0xfedf060c in memcpy () 
   from /usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1 
No symbol table info available. 
#1  0x00048060 in char_buffer_read (buffer=0x196d54, 
    in=0x196d60 "Cet-Cookie: s"..., 
    at ssl_engine_io.c:348 
No locals. 
#2  0x00048448 in ssl_io_input_read (inctx=0x196d38, 
    buf=0x196d60 "Cet-Cookie: s"..., 
    len=0xffbeafa4) at ssl_engine_io.c:561 
        wanted = 8192 
        bytes = 1533728 
        rc = 8192 
#3  0x00048714 in ssl_io_input_getline (inctx=0x196d38, 
    buf=0x196d60 "Cet-Cookie: s"..., 
    len=0xffbeafa4) at ssl_engine_io.c:712 
        pos = 0x173ea0 "" 
        status = 1666360 
        tmplen = 1 
        offset = 1533728 
(gdb) up 2 
#2  0x00048448 in ssl_io_input_read (inctx=0x196d38, 
    buf=0x196d60 "Cet-Cookie: s"..., 
    len=0xffbeafa4) at ssl_engine_io.c:561 
561 
(gdb) print *inctx 
$1 = {ssl = 0x173db8, bio_out = 0x172210, f = 0x198d68, rc = 0, 
  mode = AP_MODE_GETLINE, block = APR_BLOCK_READ, bb = 0x198d80, cbuf = { 
    length = 1, value = 0xffffffff <Address 0xffffffff out of bounds>}, 
  pool = 0x16fea8, 
  buffer = "Cet-Cookie: s"..., 
  filter_ctx = 0x17e058} 
(gdb) info locals 
wanted = 8192 
bytes = 1533728 
rc = 8192 
(gdb) print buf 
$2 = 0x196d60 "Cet-Cookie: s"..., 
(gdb) print len 
$3 = (apr_size_t *) 0xffbeafa4 
 
 
Comment 5 Joe Orton 2004-08-16 12:15:45 UTC
Ah ha, that's crucial info, thanks.

It looks like the cause is: ssl_io_input_read is called with inctx->mode ==
SPECULATIVE (this only normally happens in the proxy IIRC); ssl_io_input_read
calls char_buffer_read, which executes the case where it does:

        buffer->value = NULL;
        buffer->length = 0;

and ssl_io_input_read then screws up the inctx->cbuf for good.

            /* We want to rollback this read. */
            inctx->cbuf.value -= bytes;
            inctx->cbuf.length += bytes;

cbuf = { length = 1, value = 0xffffffff <Address 0xffffffff out of bounds>}, 

per your backtrace.

Comment 6 Joe Orton 2004-08-17 12:48:06 UTC
Created attachment 12459 [details]
proposed fix
Comment 7 Joe Orton 2004-08-17 12:49:35 UTC
The patch attached above should fix the segfaults, I'm not yet sure if this is
the cleanest or most correct fix.
Comment 8 Joe Orton 2004-08-17 21:03:21 UTC
The fix checked in should be equivalent:

http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/ssl/ssl_engine_io.c?r1=1.125&r2=1.126

This issue has possible security implications; it's been assigned CVE
CAN-2004-0751 (cve.mitre.org).  Thanks for the report and for the debugging.