Bug 51867

Summary: Apache translates a 401 to 502 from a backend proxy
Product: Apache httpd-2 Reporter: Sachin Shetty <sshetty>
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED INVALID    
Severity: critical CC: sshetty
Priority: P2    
Version: 2.3.12-beta   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Attachments: TCP Dump showing request response between Apache and the proxy

Description Sachin Shetty 2011-09-22 07:01:39 UTC
Created attachment 27557 [details]
TCP Dump showing request response between Apache and the proxy

Hi,

Our proxy setup is Apache ==> Haproxy ==> Cherry Py Server Pool

We are seeing some weird issues with Apache translating a 401 from a backend haproxy to 502 to the client. The request gets logged as 401 in haproxy logs, but apache logs it as 502 and sends a 502 back to the client. This can be easily reproduced by sending a multipart post request to Apache which forwards to haproxy and haproxy forwards it to cherry py server. Cherry py server sends a 401 back without attempting to read the post request due to some business rules.

We see the following in the apache log:

[Tue Sep 20 13:00:39 2011] [debug] mod_proxy_http.c(56): proxy: HTTP: canonicalising URL //localhost:9910/put_file
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(1525): [client 127.0.0.1] proxy: *: found reverse proxy worker for http://localhost:9910/put_file
[Tue Sep 20 13:00:39 2011] [debug] mod_proxy.c(993): Running scheme http handler (attempt 0)
[Tue Sep 20 13:00:39 2011] [debug] mod_proxy_http.c(1962): proxy: HTTP: serving URL http://localhost:9910/put_file
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2011): proxy: HTTP: has acquired connection for (*)
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2067): proxy: connecting http://localhost:9910/put_file to localhost:9910
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2193): proxy: connected /put_file to localhost:9910
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2444): proxy: HTTP: fam 2 socket created to connect to *
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2576): proxy: HTTP: connection complete to 127.0.0.1:9910 (localhost)
[Tue Sep 20 13:00:39 2011] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Tue Sep 20 13:00:39 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 127.0.0.1:9910 (localhost)
[Tue Sep 20 13:00:39 2011] [error] proxy: pass request body failed to 127.0.0.1:9910 (localhost) from 127.0.0.1 ()
[Tue Sep 20 13:00:39 2011] [debug] proxy_util.c(2029): proxy: HTTP: has released connection for (*)

It almost seems like Apache keeps sending the data to the proxy and is expecting the data (whatever is available) to be read by the proxy which does not happen since the downstream proxy has already served 401 and decided to abort

A few interesting things:
1. When I remove apache from the chain and hit haproxy directly for testing, I see a proper 401 status code as expected
2. When I post a smaller file, say 2KB, it works fine, I see a 401 status code as returned the backend proxy. But when I post a 150+ KB file, it fails with 502 even though the proxy still returned 401.

Attached tcpdump, which shows a proper HTTP Response from the backend proxy.
Comment 1 Ruediger Pluem 2011-09-22 10:49:30 UTC
This is a bug in haproxy. It needs to consume the body of requests even if it does not need them.