Bug 39499

Summary: error reading status line, reverse proxy, broken pipe
Product: Apache httpd-2 Reporter: Jan P. Sorensen <japs>
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED DUPLICATE    
Severity: normal CC: han44solo, pupilla
Priority: P2    
Version: 2.2.2   
Target Milestone: ---   
Hardware: All   
OS: Linux   

Description Jan P. Sorensen 2006-05-06 13:18:31 UTC
I am running a reverse proxy server. Since I upgraded to Apache 2.2.0
(and 2.2.2) I have had a lot of errors: "error reading status line"

The relevant part of the config file is:

<VirtualHost *:80>
    ServerName  www.ku.dk
ProxyPass             /   http://malamute.adm.ku.dk/
ProxyPassReverse      /   http://malamute.adm.ku.dk/
</VirtualHost>

With "debug" I get the following error_log:

[Sat May 06 13:52:12 2006] [debug] mod_proxy_http.c(54): proxy: HTTP: 
canonicalising URL //malamute.adm.ku.dk/net
[Sat May 06 13:52:12 2006] [debug] proxy_util.c(1378): [client 
2001:878:100:14::134] proxy: http: found worker http://malamute.adm.ku.dk/ for 
http://malamute.adm.ku.dk/net
[Sat May 06 13:52:12 2006] [debug] mod_proxy.c(756): Running scheme http 
handler (attempt 0)
[Sat May 06 13:52:12 2006] [debug] mod_proxy_http.c(1662): proxy: HTTP: 
serving URL http://malamute.adm.ku.dk/net
[Sat May 06 13:52:12 2006] [debug] proxy_util.c(1798): proxy: HTTP: has 
acquired connection for (malamute.adm.ku.dk)
[Sat May 06 13:52:12 2006] [debug] proxy_util.c(1858): proxy: connecting 
http://malamute.adm.ku.dk/net to malamute.adm.ku.dk:80
[Sat May 06 13:52:12 2006] [debug] proxy_util.c(1951): proxy: connected /net 
to malamute.adm.ku.dk:80
[Sat May 06 13:52:12 2006] [debug] proxy_util.c(2141): proxy: HTTP: connection 
complete to 130.225.126.178:80 (malamute.adm.ku.dk)
[Sat May 06 13:52:12 2006] [info] [client 130.225.126.178] (32)Broken pipe: 
core_output_filter: writing data to the network
[Sat May 06 13:52:12 2006] [error] [client 2001:878:100:14::134] proxy: error 
reading status line from remote server malamute.adm.ku.dk
[Sat May 06 13:52:12 2006] [error] [client 2001:878:100:14::134] proxy: Error 
reading from remote server returned by /net
[Sat May 06 13:52:12 2006] [debug] proxy_util.c(1816): proxy: HTTP: has 
released connection for (malamute.adm.ku.dk)

Here I have used IPv6 between the client and proxy server, and IPv4 between 
the proxy server and the backend server. However, I observe the same error 
with pure IPv4.

The error occur very often (every 2-5 minutes) during light load of the proxy 
server. During moderate load the error is very seldom (every 60-120 minutes).

Linux proxy4.adm.ku.dk 2.6.12-18mdk #1 Wed Mar 15 13:15:23 MST 2006 i686 Intel
(R) Pentium(R) 4 CPU 3.20GHz unknown GNU/Linux

Jan P. Sorensen
Comment 1 James 2006-05-07 00:49:32 UTC
I have been seeing a similar issue when using 2.2.2, worker MPM and an IIS
backend server. Everything works ok with an apache backend.

Have done some limited investication, 

Everything works fine untill the backend server closes the connection, due to a
keep-alive timeout ( hence why it works better under a higher load as the
timeout is reached less often due to mod_proxy pooling connections). Have found
the IIS closes the connection with an RST packet while apache does a fin/ack
reset handshake. When mod_proxy recv's a rst from the backend the next request
to that proxy worker causes the issue listed here. A subsequent reqest sets up a
new back end connection and things work ok again until the keep-alive time out.

I have tried this with the pre-threaded mpm and everything is fine. Have also
found the same issue with HEAD.

My solution was to set the timeout on the backend server to something stupid, in
my case a week.

I have a decent dev system in the lab atm so if needed can provide any addition
info if instructions are given.

James
Comment 2 James 2006-05-07 00:52:09 UTC
I ofc ment pre-fork MPM , sorry its late here :)
Comment 3 Marco Berizzi 2006-05-17 13:29:05 UTC
same problem here with apache 2.2.2 (on linux 2.6.17-rc4) and IIS on windows
2003sp1 + owa exchange 2003sp2
I have workarounded the problem adding 'SetEnv proxy-nokeepalive 1' to the
virtualhost section, or removing the flag 'Enable HTTP Keep-Alives' in the IIS
manager on windows 2003. Default timeout for keep-alives connections is 2
minutes (120 seconds) for windows 2003sp1

apache 1.3.35 is working *fine*
Comment 4 Marco Berizzi 2006-05-17 13:35:14 UTC
...mmm I don't know if this relevant: my envrinonment is all ipv4
Comment 5 Marco Berizzi 2006-05-17 13:59:39 UTC
I did try to set 'connection timeout' to 604800 seconds (1 week) in IIS
(back-end server) but the problem isn't resolved.
Comment 6 Marco Berizzi 2006-06-14 09:36:11 UTC
Same issue with windows 2000sp4 and IIS 4.0 on windows NT 4.0sp6a
Comment 7 Marco Berizzi 2006-06-14 10:28:01 UTC
Here is a tcpdump capture where IIS 4.0/windows nt4.0sp6a (172.16.1.78) is
resetting the connection to the apache (172.16.1.247). After that, I get 'proxy:
error reading status line from remote server' error.

19:12.981716 IP (tos 0x0, ttl 128, id 33088, offset 0, flags [DF], proto: TCP
(6), length: 40) 172.16.1.78.80 > 172.16.1.247.3355: R, cksum 0xc03b (correct),
2080897236:2080897236(0) win 0
Comment 8 Srebrenko Sehic 2006-12-22 15:25:45 UTC
You could try the fix from PR #37770. That should fix it.

Comment 9 Jan P. Sorensen 2007-01-14 05:20:37 UTC
#37770 solves the problem
Comment 10 Jan P. Sorensen 2007-01-14 05:24:44 UTC

*** This bug has been marked as a duplicate of 37770 ***
Comment 11 Marco Berizzi 2007-01-16 01:24:39 UTC
I can confirm #37770 solves the problem
Comment 12 abhishek singh 2007-07-28 02:37:23 UTC
I am using apache as a webserver and proxy loadbalancer for backed mongrels . OS
is FreeBSD 5.4 and apache 2.2.4 , i am getting error related to proxy in my
apache  error log , this is happen when server has good traffic here i am
pasting myy apache config and error logs



<VirtualHost *:80>
ServerName example.com
 
DocumentRoot /home/web/public

<Directory "/home/web/public">
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1
</Directory>

<Location />
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
</Location>


# Disable  forward proxy for security
ProxyRequests Off

# Enable reverse proxying
<Proxy *>
      Order deny,allow
      Allow from all
</Proxy>
ProxyPreserveHost on

<Proxy balancer://mongrel_cluster>

    BalancerMember http://192.168.1.110:6001 
    BalancerMember http://192.168.1.110:6002 
    BalancerMember http://192.168.1.110:6003 
    BalancerMember http://192.168.1.110:6004 
    BalancerMember http://192.168.1.110:6005 
    BalancerMember http://192.168.1.110:6006 
    BalancerMember http://192.168.1.108:6002 
    BalancerMember http://192.168.1.108:6003 
    BalancerMember http://192.168.1.108:6004 
    BalancerMember http://192.168.1.108:6005 
    BalancerMember http://192.168.1.108:6006 

</Proxy>

RewriteEngine On
#RewriteLog /var/log/httpd/rewrite_log
#RewriteLogLevel 9

#Rule For monitoring Web server from loadbalancer

RewriteRule ^/webmonitoring.html /webmonitoring.html [L]

# Check for maintenance file and redirect all requests
#  ( this is for use with Capistrano's disable_web task )

 RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]

#RewriteCond %{HTTP_HOST}  !^www.example.com [NC]
#RewriteRule ^/(.*)  http://www.example.com/$1 [R,L]

# Rewrite index to check for static
  RewriteRule ^/(.*\.(css|js|gif|png|jpg))\.v[0-9.]+$ /$1
  RewriteRule ^/$ /index.html [QSA]

# Rewrite to check for Rails cached page
#  RewriteRule ^([^.]+)$ $1.html [QSA]

# Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

# Deflate
  AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript

<DirectoryMatch "^/.*/\.svn/">
    ErrorDocument 403 /404.html
    Order allow,deny
    Deny from all
    Satisfy All
  </DirectoryMatch>

ErrorLog /var/log/httpd/httpd-error.log
CustomLog /var/log/httpd/httpd-access.log combined


</VirtualHost>

Sat Jul 28 00:32:38 2007] [error] [client xx.xx.xx.xx] proxy: error reading
status line from remote server  xx.xx.xx.xx
[Sat Jul 28 00:32:38 2007] [error] [client xx.xx.xx.xx] proxy: Error reading
from remote server returned by /embed-services/contest/51895
[Sat Jul 28 00:32:42 2007] [error] [client xx.xx.xx.xx] proxy: error reading
status line from remote server 192.168.1.x
[Sat Jul 28 00:32:42 2007] [error] [client xx.xx.xx.xx] proxy: Error reading
from remote server returned by /rss/latest
g