Bug 44105 - Core dumps when using Proxy Keepalive
Summary: Core dumps when using Proxy Keepalive
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.0.59
Hardware: Other Solaris
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Keywords: MassUpdate
Depends on:
Reported: 2007-12-19 07:35 UTC by Axel-Stephane Smorgrav
Modified: 2018-11-07 21:09 UTC (History)
0 users

Log outputs and stack trace (7.68 KB, text/plain)
2007-12-19 07:43 UTC, Axel-Stephane Smorgrav
Truss output and debug log (2.29 KB, text/plain)
2007-12-21 01:06 UTC, Axel-Stephane Smorgrav
pstack from frozen threads (3.41 KB, text/plain)
2007-12-26 00:14 UTC, Axel-Stephane Smorgrav

Note You need to log in before you can comment on or make changes to this bug.
Description Axel-Stephane Smorgrav 2007-12-19 07:35:24 UTC
When using httpd as a reverse proxy with keep-alive client connections
(KeepAlive On, nokeepalive=0), we observe frequent core dumps when accessing the
reverse proxy using browsers for which httpd is configured to accept keep-alive

This occurs even under VERY light load: one browser loading one single resource
over and over again.

The core dumps cannot be reproduced with browsers for which Apache is configured
to close the connection (nokeepalive=1) after each request. Reducing the value
of KeepAliveTimeout, or that of MaxKeepAliveRequests does not make the problem
go away.

One way to circumvent the problem however, is to set proxy-nokeepalive=1.

This bug may be related to BR 12426.
Comment 1 Axel-Stephane Smorgrav 2007-12-19 07:43:18 UTC
Created attachment 21293 [details]
Log outputs and stack trace

This attachment contains output from debug log, log_forensic as well as a stack

At the time of the core dump there was only one request (for an image)
processed by the server. KeepAlive was used on the client connection as well as
the backend connection (proxy-nokeepalive=0). The logs and the stack trace
clearly show that the core dump occurs in a function called from

I seems to me like the block to which the variable buffer points is corrupted.
AFAIU it is supposed to contain the request string but it seems to contain
garbage. This is further confirmed by the value of backasswards==1 which should
only be the case if the HTTP version is not present in the request, which it
definitively is.
Comment 2 Ruediger Pluem 2007-12-20 12:30:17 UTC
(In reply to comment #1)

> I seems to me like the block to which the variable buffer points is corrupted.

A lot more seems to be corrupted:

0xff10d09c in apr_table_do (comp=0x1, rec=0xf82f9a58, t=0x7b400)
    at apr_tables.c:885

comp should be pointing to a function and by no means should ever be 0x1! The
function pointer is not supplied in a variable but as a constant.

 0x00035c28 in ap_proxy_http_process_response (p=0x304ea8, r=0x30ef08, 
    p_conn=0x363e38, origin=0x305608, backend=0x305400, conf=0x1e9988, 
    bb=0x363de8, server_portstr=0xf82fbaf0 "") at proxy_http.c:1656

The line number does not match at all (or you have a patched version of the proxy)

Please try the latest 2.0.x version with a vanilla build and let me know if the
problem still persists.
Comment 3 Axel-Stephane Smorgrav 2007-12-21 01:04:43 UTC
(In reply to comment #2)
> The line number does not match at all (or you have a patched version of the proxy)

I forgot that I did patch mod_proxy with

> Please try the latest 2.0.x version with a vanilla build and let me know if the
> problem still persists.

I made a new test with a vanilla build of mod_proxy 2.0.59. This time the server
did not crash (dump core), but the request processing appears to freeze whenever
keepalive is used (front-end + back-end). When I disable proxy keepalive
(proxy-nokeepalive=1), it serves the proxied resources normally.

I'll attach debug and truss output
Comment 4 Axel-Stephane Smorgrav 2007-12-21 01:06:48 UTC
Created attachment 21299 [details]
Truss output and debug log
Comment 5 Ruediger Pluem 2007-12-21 08:06:28 UTC
(In reply to comment #4)
> Created an attachment (id=21299) [edit]
> Truss output and debug log 

The truss does not really help. If it freezes try to get a backtrace either via
pstack or gdb.
Comment 6 Axel-Stephane Smorgrav 2007-12-26 00:14:47 UTC
Created attachment 21317 [details]
pstack from frozen threads

Here is the output from pstack.

It shows two threads. This is because we use two VH to reverse proxy the
request; one for SSL termination and content compression, and the other for
determining where to route the request and cache the contents.

For debug purposes we have however disabled caching.
Comment 7 Axel-Stephane Smorgrav 2007-12-26 00:47:01 UTC
(In reply to comment #6)

I guess I spoke too soon about the two threads...

Anyway I simplified the configuration so that there is only one VH involved in
serving any request. The behaviour is still the same and so is the pstack.
Comment 8 William A. Rowe Jr. 2018-11-07 21:09:27 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.