|Summary:||mod_proxy fails to send FIN response when a FIN is received from a backend|
|Product:||Apache httpd-2||Reporter:||Gregory Boyce <gregory.boyce>|
|Component:||mod_proxy||Assignee:||Apache HTTPD Bugs Mailing List <bugs>|
|Severity:||major||CC:||drop, mdlockwood, orion, taffy-tyler6464|
|Bug Depends on:|
|Attachments:||close_wait comparsion 1.3 vs. 2.4|
Description Gregory Boyce 2011-02-18 14:11:03 UTC
I'm currently responsible for managing some reverse proxy servers using apache and mod_proxy, and I stumbled across an apparent bug that I'm hoping someone might have some insights into. Mod_proxy is configured to frontend various webservers, typically with persistent connections enabled on both the frontend and on the backend. What I've discovered is that when there is an established connection to the backend and the remote side of the connection sends a FIN, mod_proxy gets into a bad state. For some reason we never send the corresponding FIN to finish closing the connection. Connections where backend persistent connections are disabled work fine. I replicated this on my Ubuntu desktop running Apache 2.2.16 with a very simple proxy configuration: ProxyPass /apache/ http://220.127.116.11/ ProxyPassReverse /apache/ http://apache.com/ I issue a curl against http://localhost/apache/ and I see the following in a packet capture: 13:56:52.377520 IP 192.168.1.57.58103 > 18.104.22.168.80: Flags [S], seq 3154899214, win 5840, options [mss 1460,sackOK,TS val 465793064 ecr 0,nop,wscale 7], length 0 13:56:52.428168 IP 22.214.171.124.80 > 192.168.1.57.58103: Flags [S.], seq 1968453344, ack 3154899215, win 5840, options [mss 1460,nop,nop,sackOK], length 0 13:56:52.428210 IP 192.168.1.57.58103 > 126.96.36.199.80: Flags [.], ack 1, win 5840, length 0 13:56:52.428453 IP 192.168.1.57.58103 > 188.8.131.52.80: Flags [P.], seq 1:266, ack 1, win 5840, length 265 13:56:52.480281 IP 184.108.40.206.80 > 192.168.1.57.58103: Flags [.], ack 266, win 6432, length 0 13:56:52.484575 IP 220.127.116.11.80 > 192.168.1.57.58103: Flags [P.], seq 1:648, ack 266, win 6432, length 647 13:56:52.484593 IP 192.168.1.57.58103 > 18.104.22.168.80: Flags [.], ack 648, win 7117, length 0 Interesting part comes after a few seconds: 13:56:57.484840 IP 22.214.171.124.80 > 192.168.1.57.58103: Flags [F.], seq 648, ack 266, win 6432, length 0 13:56:57.519404 IP 192.168.1.57.58103 > 126.96.36.199.80: Flags [.], ack 649, win 7117, length 0 In netstat I see an apache thread in a CLOSE_WAIT state: tcp 1 0 192.168.1.57:58103 188.8.131.52:80 CLOSE_WAIT apachectl-status shows one of the threads in a "W" Sending Reply status. I haven't found any existing bugs on this, but I do see an old e-mail with a similar behavior for mod_proxy's ftp data connection back in 2000: http://mail-archives.apache.org/mod_mbox/www-apache-bugdb/200001.mbox/%3C20000110120507.firstname.lastname@example.org%3E
Comment 1 Eric Covener 2011-02-18 14:30:01 UTC
Isn't this just the state the connection sits in until a subsequent request to the proxy finds the connection in the pool and tries to use it? The request that established the connection has moved on by the time the origin server is performing a keepalive close.
Comment 2 Gregory Boyce 2011-02-18 14:39:36 UTC
(In reply to comment #1) > Isn't this just the state the connection sits in until a subsequent request to > the proxy finds the connection in the pool and tries to use it? The request > that established the connection has moved on by the time the origin server is > performing a keepalive close. I'm not quite sure what you mean. The client request that triggered the backend connection has definitely been responded to and closed. The backend connection itself is was kept alive until the backend webserver's timeout was reached, and it attempted to close the connection by sending a FIN. I would expect that Apache would then send its own FIN and open a new connection for subsequent requests.
Comment 3 Gregory Boyce 2011-02-18 14:46:32 UTC
Ah, I see. It appears that Apache does eventually send a FIN packet the next time that the specific process attempts to establish a new connection. Until that point, both ends of the connection are left in a half closed state until either the next request triggers the FIN and a subsequent SYN, or the servers timeout the connection. This ended up biting us when a load balancer had a very long timeout value (30 minutes) and we were apparently not sending enough requests through in order to flush out the FINs. Eventually a request would attempt to reuse the same source port and the load balancer wouldn't respond to the SYN requests since it was still waiting on a FIN from us on that source port. We've worked around it for now by dropping the timeout on the load balancer.
Comment 4 Adam Woodworth 2011-06-30 15:23:10 UTC
Has anyone looked into this problem? Is a fix available? This is a critical bug with mod_proxy. We've now seen this problem 3 times at different sites. When using backend keepalives mod_proxy needs to be fixed to send a FIN to the backend server (i.e. close the connection) when it gets a FIN from the backend server. Otherwise the connection is half opened and in many cases a firewall is tracking that and will not allow that same source port to be reused until a FIN is received or until the firewall's timeout expires for that type of tracking.
Comment 5 Adam Woodworth 2011-07-07 16:06:09 UTC
This bug is made worse by the fact that net.netfilter.nf_conntrack_tcp_timeout_close_wait in the Linux kernel (at least on some versions) has a default value of 60 seconds. This value is used when iptables is tracking connections, and it is how long conntrack will wait for a responding FIN after seeing a FIN from one side. In this case, with the default value, when Apache finally tries to close a half-closed connection, if it has been longer than 60 seconds then Apache will not be able to send its FIN, making this problem worse. Increasing the value, of course, is an option, but probably not often done or even known about.
Comment 6 Jim Jagielski 2012-11-01 14:32:25 UTC
Can you confirm that the issue still exists in 2.4.x?
Comment 7 sebmoule 2012-11-13 12:53:39 UTC
Is there any correction for this problem ? does apache 2.4 correct it ?
Comment 8 William A. Rowe Jr. 2013-03-06 21:32:00 UTC
This is not needinfo, as Eric properly identified the condition and there is no change I'm aware of to the 2.4 pools which would close them early. That said, there is no clear fix because those connections are already gone. The workaround, AIUI (as noted in comment 3) is to ensure the backend keepalive timeouts (and intervening firewalls, routers and load balancers) are always longer than httpd's proxy keepalive timeouts. Remaining _WAIT connections should be small enough in number to handle gracefully. Better yet; since proxy workers should never need recycling, use an unlimited keepalive window where sensible. Perhaps a proxy pool management thread could park unused workers and respond to the signalled poll condition.
Comment 9 William A. Rowe Jr. 2013-03-06 21:47:04 UTC
As a short-term band-aid, should all disconnected proxy connections which hadn't reached their timeout emit a warning like 'Proxy connection to %s unexpectedly disconnected; the keepalive timeout of that backend server or an intermediate proxy, router, load balancer or firewall appliance is too short, all must be greater than %d seconds as configured in httpd' ?
Comment 10 Orion Poplawski 2014-05-15 20:53:33 UTC
Adam's not in comment #5 applies to just standard systems as well where the net.ipv4.tcp_fin_timeout defaults to 60 seconds. So when apache tries to send the final FIN, we get a flood of FIN and RST packets due to tcp retransmission: 29 5.076937000 BACKEND -> PROXY TCP 66 8141 > 59404 [FIN, ACK] Seq=7266 Ack=2506 Win=35968 Len=0 TSval=372987155 TSecr=373232386 30 5.116993000 PROXY -> BACKEND TCP 66 59404 > 8141 [ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373237391 TSecr=372987155 616 168.070814000 PROXY -> BACKEND TCP 66 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373400344 TSecr=372987155 617 168.070868000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 632 168.274981000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373400549 TSecr=372987155 633 168.275009000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 634 168.685140000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373400959 TSecr=372987155 635 168.685196000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 636 169.504941000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373401779 TSecr=372987155 637 169.504968000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 640 171.144951000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373403419 TSecr=372987155 641 171.144982000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 644 174.454315000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373406728 TSecr=372987155 645 174.454348000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 673 181.013929000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373413288 TSecr=372987155 674 181.013955000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 732 194.133990000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373426408 TSecr=372987155 733 194.134013000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0 877 220.374086000 PROXY -> BACKEND TCP 66 [TCP Retransmission] 59404 > 8141 [FIN, ACK] Seq=2506 Ack=7267 Win=32000 Len=0 TSval=373452648 TSecr=372987155 878 220.374149000 BACKEND -> PROXY TCP 54 8141 > 59404 [RST] Seq=7267 Win=0 Len=0
Comment 11 Andy Wang 2014-09-22 16:18:19 UTC
Looking at the mod_proxy documentation there's this little tidbit: ttl can be used to avoid using a connection which is subject to closing because of the backend server's keep-alive timeout. So wouldn't that be the proper configurable solution?
Comment 12 Hendrik Harms 2016-09-19 07:06:50 UTC
Created attachment 34270 [details] close_wait comparsion 1.3 vs. 2.4 Just an example: upgrading from 1.3.42 to 2.4.23-prefork increases TCP CLOSE_WAIT dramatically. All the close_waits are belonging to mod_proxy(http) backend connections.
Comment 13 Yann Ylavic 2018-03-29 09:12:53 UTC
The main difference between 1.3 and 2.2+ (possibly 2.0 too) is that backend connections are managed/reused independently from the clients' ones. In 1.3 the backends and clients connections were closed together (at worst when client's KeepAlive expires), this is not the case anymore. Pros: big performance win (think of TLS and/or TCP-3WAY handshakes saved, which quite matters on heavy load). Cons: ttl fine tuning (according to / lower than the backend's KeepAliveTimeout), both for keeping CLOSE_WAIT sockets low AND avoid reuse race conditions.
Comment 14 William A. Rowe Jr. 2018-11-07 21:09:29 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.