Bug 44079

Summary: mod_proxy_http randomly fails with 'Error reading from remote server' when connection to backend server is non-SSL
Product: Apache httpd-2 Reporter: Todor Zviskov <todorz>
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED DUPLICATE    
Severity: major    
Priority: P2    
Version: 2.2.3   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description Todor Zviskov 2007-12-14 08:52:31 UTC
mod_proxy_http randomly returns a proxy error when the proxy request to the
backend server is non-SSL. The apache server is configured with both SSL and
non-SSL but the error randomly occurs no matter if the request is http or https.
With proxy configuration:

        ProxyPass /bugs http://internal.server.com/bugs
        ProxyPassReverse /bugs http://internal.server.com/bugs


Apache randomly returns 'Error reading from remote server':

        [Thu Dec 13 15:40:47 2007] [debug] mod_proxy_http.c(54): proxy: HTTP:
canonicalising URL //internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:47 2007] [debug] proxy_util.c(1378): [client
192.168.1.131] proxy: http: found worker http://internal.server.com/bugs for
http://internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:47 2007] [debug] mod_proxy.c(756): Running scheme http
handler (attempt 0)
        [Thu Dec 13 15:40:47 2007] [debug] mod_proxy_ajp.c(493): proxy: AJP:
declining URL http://internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:47 2007] [debug] mod_proxy_http.c(1662): proxy: HTTP:
serving URL http://internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:47 2007] [debug] proxy_util.c(1798): proxy: HTTP: has
acquired connection for (internal.server.com)
        [Thu Dec 13 15:40:47 2007] [debug] proxy_util.c(1858): proxy: connecting
http://internal.server.com/bugs/heartbeat.asp to internal.server.com:80
        [Thu Dec 13 15:40:47 2007] [debug] proxy_util.c(1951): proxy: connected
/bugs/heartbeat.asp to internal.server.com:80
        [Thu Dec 13 15:40:47 2007] [debug] proxy_util.c(2141): proxy: HTTP:
connection complete to 192.168.1.131:80 (internal.server.com)
        [Thu Dec 13 15:40:47 2007] [info] [client 192.168.1.131] (32)Broken
pipe: core_output_filter: writing data to the network
        [Thu Dec 13 15:40:47 2007] [error] [client 192.168.1.131] proxy: error
reading status line from remote server internal.server.com
        [Thu Dec 13 15:40:47 2007] [error] [client 192.168.1.131] proxy: Error
reading from remote server returned by /bugs/heartbeat.asp
        [Thu Dec 13 15:40:47 2007] [debug] proxy_util.c(1816): proxy: HTTP: has
released connection for (internal.server.com)


More often, the request is processed correctly:

        [Thu Dec 13 15:40:32 2007] [debug] mod_proxy_http.c(54): proxy: HTTP:
canonicalising URL //internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(1378): [client
192.168.1.131] proxy: http: found worker http://internal.server.com/bugs for
http://internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:32 2007] [debug] mod_proxy.c(756): Running scheme http
handler (attempt 0)
        [Thu Dec 13 15:40:32 2007] [debug] mod_proxy_ajp.c(493): proxy: AJP:
declining URL http://internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:32 2007] [debug] mod_proxy_http.c(1662): proxy: HTTP:
serving URL http://internal.server.com/bugs/heartbeat.asp
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(1798): proxy: HTTP: has
acquired connection for (internal.server.com)
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(1858): proxy: connecting
http://internal.server.com/bugs/heartbeat.asp to internal.server.com:80
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(1951): proxy: connected
/bugs/heartbeat.asp to internal.server.com:80
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(2045): proxy: HTTP: fam
2 socket created to connect to internal.server.com
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(2141): proxy: HTTP:
connection complete to 192.168.1.131:80 (internal.server.com)
        [Thu Dec 13 15:40:32 2007] [debug] mod_proxy_http.c(1448): proxy: start
body send
        [Thu Dec 13 15:40:32 2007] [debug] mod_proxy_http.c(1537): proxy: end
body send
        [Thu Dec 13 15:40:32 2007] [debug] proxy_util.c(1816): proxy: HTTP: has
released connection for (internal.server.com)


It's interesting to note that when the error occurs, there's no socket created.
When the connection to the backend server is SSL:

        ProxyPass /bugs https://internal.server.com/bugs
        ProxyPassReverse /bugs https://internal.server.com/bugs


everything works correctly 100% of the time. From what I know, mod_proxy cannot
handle persistent SSL backend connections. Is it possible that the non-SSL
backend connections are persistent and that's causing all this? I've tried quite
a few things, including  'SetEnv force-proxy-request-1.0 1 SetEnv
proxy-nokeepalive 1' without luck. Apache 1.x didn't had this issue.
Comment 1 Nick Kew 2007-12-14 12:29:41 UTC
Whilst this is a long-standing issue, your report is useful as it tends to
support Rudiger's diagnosis of what causes it.  Thank you.

*** This bug has been marked as a duplicate of 37770 ***
Comment 2 Ruediger Pluem 2007-12-14 14:10:23 UTC
(In reply to comment #0)

> 
> everything works correctly 100% of the time. From what I know, mod_proxy cannot
> handle persistent SSL backend connections. Is it possible that the non-SSL
> backend connections are persistent and that's causing all this? I've tried 

Yes, this is exactly the reason. SSL backend connections are not persistent
whereas non-SSL connections are.


> quite a few things, including  'SetEnv force-proxy-request-1.0 1 SetEnv
> proxy-nokeepalive 1' without luck. Apache 1.x didn't had this issue.

proxy-nokeepalive 1 should really help as it ensures that the backend connection
gets closed like in the SSL case. I assume you placed

SetEnv proxy-nokeepalive 1

in the wrong place of your config. Where did you place it?