When using httpd as a reverse proxy with keep-alive client connections (KeepAlive On, nokeepalive=0), we observe frequent core dumps when accessing the reverse proxy using browsers for which httpd is configured to accept keep-alive connections. This occurs even under VERY light load: one browser loading one single resource over and over again. The core dumps cannot be reproduced with browsers for which Apache is configured to close the connection (nokeepalive=1) after each request. Reducing the value of KeepAliveTimeout, or that of MaxKeepAliveRequests does not make the problem go away. One way to circumvent the problem however, is to set proxy-nokeepalive=1. This bug may be related to BR 12426.
Created attachment 21293 [details] Log outputs and stack trace This attachment contains output from debug log, log_forensic as well as a stack trace. At the time of the core dump there was only one request (for an image) processed by the server. KeepAlive was used on the client connection as well as the backend connection (proxy-nokeepalive=0). The logs and the stack trace clearly show that the core dump occurs in a function called from ap_proxy_http_handler. I seems to me like the block to which the variable buffer points is corrupted. AFAIU it is supposed to contain the request string but it seems to contain garbage. This is further confirmed by the value of backasswards==1 which should only be the case if the HTTP version is not present in the request, which it definitively is.
(In reply to comment #1) > > I seems to me like the block to which the variable buffer points is corrupted. A lot more seems to be corrupted: 0xff10d09c in apr_table_do (comp=0x1, rec=0xf82f9a58, t=0x7b400) at apr_tables.c:885 comp should be pointing to a function and by no means should ever be 0x1! The function pointer is not supplied in a variable but as a constant. 0x00035c28 in ap_proxy_http_process_response (p=0x304ea8, r=0x30ef08, p_conn=0x363e38, origin=0x305608, backend=0x305400, conf=0x1e9988, bb=0x363de8, server_portstr=0xf82fbaf0 "") at proxy_http.c:1656 The line number does not match at all (or you have a patched version of the proxy) Please try the latest 2.0.x version with a vanilla build and let me know if the problem still persists.
(In reply to comment #2) > The line number does not match at all (or you have a patched version of the proxy) I forgot that I did patch mod_proxy with http://people.apache.org/~colm/httpd-2.0-reverse-proxy-cookie.patch > > Please try the latest 2.0.x version with a vanilla build and let me know if the > problem still persists. I made a new test with a vanilla build of mod_proxy 2.0.59. This time the server did not crash (dump core), but the request processing appears to freeze whenever keepalive is used (front-end + back-end). When I disable proxy keepalive (proxy-nokeepalive=1), it serves the proxied resources normally. I'll attach debug and truss output
Created attachment 21299 [details] Truss output and debug log
(In reply to comment #4) > Created an attachment (id=21299) [edit] > Truss output and debug log > The truss does not really help. If it freezes try to get a backtrace either via pstack or gdb.
Created attachment 21317 [details] pstack from frozen threads Here is the output from pstack. It shows two threads. This is because we use two VH to reverse proxy the request; one for SSL termination and content compression, and the other for determining where to route the request and cache the contents. For debug purposes we have however disabled caching.
(In reply to comment #6) I guess I spoke too soon about the two threads... Anyway I simplified the configuration so that there is only one VH involved in serving any request. The behaviour is still the same and so is the pstack.
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.