Overview Description: Intermittent segmentation faults occur in char_buffer_read at ssl_engine_io.c:348 when using a RewriteRule to do reverse proxying to an SSL origin server running IIS. Steps to Reproduce: 1) Set up an IIS server using SSL and running eRoom 6. 2) Add the following directives to httpd.conf: Listen 47290 SSLProxyEngine on RewriteEngine on RewriteRule /(.*) https://some.eroom6.iis.server.com/$1 [P] 3) Visit a URL similar to the following: http://reverse.proxy.com:47290/eRoomASP/CookieTest.asp?facility=facility&URL=%2FeRoom%2FFacility%2FRoom%2F0_4242 If that doesn't cause the segfault, click around for a while. (Yes, reverse proxying from non-SSL to SSL is not a good idea, but it keeps the example simpler.) Actual Results: Segmentation fault in error_log: [Thu Jul 15 19:38:36 2004] [notice] child pid 42 exit signal Segmentation fault (11), possible coredump in /usr/local/httpd-2.0.50 Build Date & Platform: 2004-07-14 build on SunOS 5.8 SUNW,UltraAX-i2 Additional Information: Here is a stack trace from gdb: #0 0xfef5060c in memcpy () from /usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1 #1 0xfeafef54 in char_buffer_read (buffer=0x1649ac, in=0x2000 <Address 0x2000 out of bounds>, inl=8192) at ssl_engine_io.c:348 #2 0xfeaff388 in ssl_io_input_read (inctx=0x164990, buf=0x1649b8 "Content-Length: 121\r\nCort/Martonia/0_2615\r\nContent-Length: 121\r\nCent-Length: 121\r\nCmeport/Martonia/0_2615\r\nContent-Length: 121\r\nCmeport/Martonia/0_2615\r\nContent-Length: 121\r\nCrtonia/0_2615\r\nContent-Le"..., len=0xffbea8cc) at ssl_engine_io.c:561 #3 0xfeaff624 in ssl_io_input_getline (inctx=0x164990, buf=0x1649b8 "Content-Length: 121\r\nCort/Martonia/0_2615\r\nContent-Length: 121\r\nCent-Length: 121\r\nCmeport/Martonia/0_2615\r\nContent-Length: 121\r\nCmeport/Martonia/0_2615\r\nContent-Length: 121\r\nCrtonia/0_2615\r\nContent-Le"..., len=0xffbea944) at ssl_engine_io.c:712 #4 0xfeb00118 in ssl_io_filter_input (f=0x1669c0, bb=0x158f98, mode=4290685252, block=APR_BLOCK_READ, readbytes=0) at ssl_engine_io.c:1226 #5 0x42978 in ap_get_brigade (next=0x1669c0, bb=0x158f98, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:474 #6 0x4aab4 in net_time_filter (f=0x158e20, b=0x158f98, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at core.c:3600 #7 0x42978 in ap_get_brigade (next=0x158e20, bb=0x158f98, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0) at util_filter.c:474 #8 0x43e0c in ap_rgetline_core (s=0xffbeab94, n=8192, read=0xffbeab90, r=0x1671b8, fold=1, bb=0x158f98) at protocol.c:214 #9 0x441a4 in ap_getline (s=0xffbeccd8 "Content-Length", n=8192, r=0x1671b8, fold=1) at protocol.c:478 #10 0xfe8552d4 in ap_proxy_read_headers (r=0x1821d0, rr=0x1671b8, buffer=0xffbeccd8 "Content-Length", size=8192, c=0x1671b8) at proxy_util.c:457 #11 0xfe833014 in ap_proxy_http_process_response (p=0x157960, r=0x1821d0, p_conn=0x157ee8, origin=0x158168, backend=0x157f00, conf=0xf0268, bb=0x157e98, server_portstr=0xffbeed68 ":47290") at proxy_http.c:755 #12 0xfe833ba4 in ap_proxy_http_handler (r=0x1821d0, conf=0xf0268, url=0x158038 "/eRoomASP/CookieTest.asp?facility=memeport&URL=%2FeRoom%2Fmemeport%2FMartonia%2F0_2615", proxyname=0x0, proxyport=60776) at proxy_http.c:1121 #13 0xfe85435c in proxy_run_scheme_handler (r=0x1821d0, conf=0xf0268, url=0x1839ce "https://eroomhost.aaa.bbb.com/eRoomASP/CookieTest.asp?facility=memeport&URL=%2FeRoom%2Fmemeport%2FMartonia%2F0_2615", proxyhost=0x0, proxyport=0) at mod_proxy.c:1113 #14 0xfe852ed8 in proxy_handler (r=0x1821d0) at mod_proxy.c:418 #15 0x359a8 in ap_run_handler (r=0x1821d0) at config.c:151 #16 0x35fa4 in ap_invoke_handler (r=0x1821d0) at config.c:358 #17 0x32c44 in ap_process_request (r=0x1821d0) at http_request.c:246 #18 0x2df14 in ap_process_http_connection (c=0x157a70) at http_core.c:250 #19 0x40090 in ap_run_process_connection (c=0x157a70) at connection.c:42 #20 0x403a4 in ap_process_connection (c=0x157a70, csd=0x157998) at connection.c:175 #21 0x3422c in child_main (child_num_arg=5) at prefork.c:609 #22 0x343ac in make_child (s=0x8edb0, slot=5) at prefork.c:703 #23 0x345fc in perform_idle_server_maintenance (p=0x8c690) at prefork.c:838 #24 0x34a34 in ap_mpm_run (_pconf=0x0, plog=0x63400, s=0x83000) at prefork.c:1039 #25 0x3ad44 in main (argc=3, argv=0xffbef4ac) at main.c:617
Does this still happen if you take mod_rewrite out of the setup and use ProxyPass instead?
Yes, I still get what seems to be the same segfault (according to gdb) if I replace these config lines: RewriteEngine on RewriteRule /(.*) https://some.eroom6.iis.server.com/$1 [P] with this one: ProxyPass / https://some.eroom6.iis.server.com/
This backtrace is a little strange: #1 0xfeafef54 in char_buffer_read (buffer=0x1649ac, in=0x2000 <Address 0x2000 out of bounds>, inl=8192) at ssl_engine_io.c:348 which means either stack corruption by memcpy or in got passed in as a pointer to address 8192 somehow. Can you, from gdb against a core dump, check: up 2 (into ssl_io_input_read) print *inctx info locals print buf print len
Hi Joe, I do suffer from the same bug. Attached is the info you requested. If you need additional testing, please say so. I cleared some info from the contents of buffer to not disclose critical information. I hope that's ok and doesn't hinder your debugging process. Regards, Stephan Program received signal SIGSEGV, Segmentation fault. 0xfedf060c in memcpy () from /usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1 (gdb) where full #0 0xfedf060c in memcpy () from /usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1 No symbol table info available. #1 0x00048060 in char_buffer_read (buffer=0x196d54, in=0x196d60 "Cet-Cookie: s"..., at ssl_engine_io.c:348 No locals. #2 0x00048448 in ssl_io_input_read (inctx=0x196d38, buf=0x196d60 "Cet-Cookie: s"..., len=0xffbeafa4) at ssl_engine_io.c:561 wanted = 8192 bytes = 1533728 rc = 8192 #3 0x00048714 in ssl_io_input_getline (inctx=0x196d38, buf=0x196d60 "Cet-Cookie: s"..., len=0xffbeafa4) at ssl_engine_io.c:712 pos = 0x173ea0 "" status = 1666360 tmplen = 1 offset = 1533728 (gdb) up 2 #2 0x00048448 in ssl_io_input_read (inctx=0x196d38, buf=0x196d60 "Cet-Cookie: s"..., len=0xffbeafa4) at ssl_engine_io.c:561 561 (gdb) print *inctx $1 = {ssl = 0x173db8, bio_out = 0x172210, f = 0x198d68, rc = 0, mode = AP_MODE_GETLINE, block = APR_BLOCK_READ, bb = 0x198d80, cbuf = { length = 1, value = 0xffffffff <Address 0xffffffff out of bounds>}, pool = 0x16fea8, buffer = "Cet-Cookie: s"..., filter_ctx = 0x17e058} (gdb) info locals wanted = 8192 bytes = 1533728 rc = 8192 (gdb) print buf $2 = 0x196d60 "Cet-Cookie: s"..., (gdb) print len $3 = (apr_size_t *) 0xffbeafa4
Ah ha, that's crucial info, thanks. It looks like the cause is: ssl_io_input_read is called with inctx->mode == SPECULATIVE (this only normally happens in the proxy IIRC); ssl_io_input_read calls char_buffer_read, which executes the case where it does: buffer->value = NULL; buffer->length = 0; and ssl_io_input_read then screws up the inctx->cbuf for good. /* We want to rollback this read. */ inctx->cbuf.value -= bytes; inctx->cbuf.length += bytes; cbuf = { length = 1, value = 0xffffffff <Address 0xffffffff out of bounds>}, per your backtrace.
Created attachment 12459 [details] proposed fix
The patch attached above should fix the segfaults, I'm not yet sure if this is the cleanest or most correct fix.
The fix checked in should be equivalent: http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/ssl/ssl_engine_io.c?r1=1.125&r2=1.126 This issue has possible security implications; it's been assigned CVE CAN-2004-0751 (cve.mitre.org). Thanks for the report and for the debugging.