Bug 51867 - Apache translates a 401 to 502 from a backend proxy
Summary: Apache translates a 401 to 502 from a backend proxy
Status: RESOLVED INVALID
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.3.12-beta
Hardware: PC Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-09-22 07:01 UTC by Sachin Shetty
Modified: 2011-09-22 10:49 UTC (History)
1 user (show)



Attachments
TCP Dump showing request response between Apache and the proxy (16.43 KB, application/x-zip-compressed)
2011-09-22 07:01 UTC, Sachin Shetty
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sachin Shetty 2011-09-22 07:01:39 UTC
Created attachment 27557 [details]
TCP Dump showing request response between Apache and the proxy

Hi,

Our proxy setup is Apache ==> Haproxy ==> Cherry Py Server Pool

We are seeing some weird issues with Apache translating a 401 from a backend haproxy to 502 to the client. The request gets logged as 401 in haproxy logs, but apache logs it as 502 and sends a 502 back to the client. This can be easily reproduced by sending a multipart post request to Apache which forwards to haproxy and haproxy forwards it to cherry py server. Cherry py server sends a 401 back without attempting to read the post request due to some business rules.

We see the following in the apache log:

[Tue Sep 20 13:00:39 2011] [debug] mod_proxy_http.c(56): proxy: HTTP: canonicalising URL //localhost:9910/put_file
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(1525): [client 127.0.0.1] proxy: *: found reverse proxy worker for http://localhost:9910/put_file
[Tue Sep 20 13:00:39 2011] [debug] mod_proxy.c(993): Running scheme http handler (attempt 0)
[Tue Sep 20 13:00:39 2011] [debug] mod_proxy_http.c(1962): proxy: HTTP: serving URL http://localhost:9910/put_file
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2011): proxy: HTTP: has acquired connection for (*)
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2067): proxy: connecting http://localhost:9910/put_file to localhost:9910
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2193): proxy: connected /put_file to localhost:9910
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2444): proxy: HTTP: fam 2 socket created to connect to *
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2576): proxy: HTTP: connection complete to 127.0.0.1:9910 (localhost)
[Tue Sep 20 13:00:39 2011] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Tue Sep 20 13:00:39 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 127.0.0.1:9910 (localhost)
[Tue Sep 20 13:00:39 2011] [error] proxy: pass request body failed to 127.0.0.1:9910 (localhost) from 127.0.0.1 ()
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2029): proxy: HTTP: has released connection for (*)

It almost seems like Apache keeps sending the data to the proxy and is expecting the data (whatever is available) to be read by the proxy which does not happen since the downstream proxy has already served 401 and decided to abort

A few interesting things:
1. When I remove apache from the chain and hit haproxy directly for testing, I see a proper 401 status code as expected
2. When I post a smaller file, say 2KB, it works fine, I see a 401 status code as returned the backend proxy. But when I post a 150+ KB file, it fails with 502 even though the proxy still returned 401.

Attached tcpdump, which shows a proper HTTP Response from the backend proxy.
Comment 1 Ruediger Pluem 2011-09-22 10:49:30 UTC
This is a bug in haproxy. It needs to consume the body of requests even if it does not need them.