Bug 65689 - proxy connections to target left in CLOSE_WAIT state
Summary: proxy connections to target left in CLOSE_WAIT state
Status: RESOLVED DUPLICATE of bug 65662
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.4.51
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-11-19 14:39 UTC by Lothar
Modified: 2021-11-23 17:28 UTC (History)
1 user (show)



Attachments
Trace8 Log filtered by one PID, serving the Request IP-Addresses/hostnames anonymised ( (50.85 KB, text/plain)
2021-11-23 08:12 UTC, Lothar
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Lothar 2021-11-19 14:39:28 UTC
Hi all,
since we updated apache from 2.4.46 to 2.4.51 we have a lot of open connections to destination hanging in "CLOSE_WAIT" state.

The connection is closed by target after a 10 secs. with a TCP RST.
Which causes alerts in our monitoring system.
I tried to disable keepalive with
      SetEnv proxy-nokeepalive 1
We are using this config:
    <IfModule mod_proxy.c>
        ProxyRequests On
        AllowCONNECT 443
        SetEnv proxy-nokeepalive 1
        <Proxy "*">
                Require all granted
        </Proxy>
    </IfModule>
The Client is initiation a connection via method CONNECT:
CONNECT www.example.com:443 HTTP/1.1
Somehow disable keepalive is not working.
Also the behavior has changed, with apache 2.4.46 the connection was closed smoothly:
apache -> target ACK
target -> apache FIN,ACK
apache -> target ACK
apache -> target FIN,ACK

With apache 2.4.51 the last FIN,ACK is not sent anymore but we receive 
target -> apache RST, ACK after 10 sec.

Could this be fixed, or is there any workaround ( like disable keepalive)

Thanks a lot!
Kind Regards, Lothar
Comment 1 Piotr Staszeski 2021-11-22 14:48:24 UTC
pstack 27036
#0  0x00007f4e0ac060c3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x00007f4e0b758a59 in ?? ()
#2  0x00007ffcdd3b13fc in ?? ()
#3  0x00007ffcdd3b14a4 in ?? ()
#4  0x00007ffcdd3b14a8 in ?? ()
#5  0x0000000001bb9b98 in ?? ()
#6  0x0000000001bb9cd8 in ?? ()
#7  0x0000000001bb9b80 in ?? ()
#8  0xffffffffffffffff in ?? ()
#9  0x00007ffcdd3b14a4 in ?? ()
#10 0x0000000001bb9ca8 in ?? ()
#11 0x0000000001bb9cd8 in ?? ()
#12 0x00007ffcdd3b14a8 in ?? ()
#13 0x00007f4e092b1721 in ?? ()
#14 0x0000000001b99a98 in ?? ()
#15 0x0000000001bace78 in ?? ()
#16 0x0000000001b994c0 in ?? ()
#17 0x00007ffcdd3b1540 in ?? ()
#18 0x0000000001bb9cd8 in ?? ()
#19 0x00007f4e0b753283 in ?? ()
#20 0x0000000001bb7e20 in ?? ()
#21 0x0000000001b994c0 in ?? ()
#22 0x0000000001bb9ca0 in ?? ()
#23 0x0000000001bb9b80 in ?? ()
#24 0x0000000001bb9b38 in ?? ()
#25 0xffffffffffffffff in ?? ()
#26 0x00007ffcdd3b14a4 in ?? ()
#27 0x00007ffcdd3b14a8 in ?? ()
#28 0x0000000001b99040 in ?? ()
#29 0x0000000001bb9c40 in ?? ()
#30 0x0000000001b99040 in ?? ()
#31 0x0000000001bb9af8 in ?? ()
#32 0x0000000001b994c0 in ?? ()
#33 0x0000000001b99040 in ?? ()
#34 0x00007ffcdd3b15b0 in ?? ()
#35 0x0000000001bb7e20 in ?? ()
#36 0x0000000000440910 in ?? () at util_filter.c:593
#37 0x00007f4e09094f35 in ?? ()
#38 0x0000000001b7bc68 in ?? ()
#39 0x00007f4e0b7477d7 in ?? ()
#40 0x0000000000000000 in ?? ()
Comment 2 Piotr Staszeski 2021-11-22 14:50:40 UTC
(In reply to Piotr Staszeski from comment #1)
> pstack 27036

missclick - @Lothar do You also see related FIN_WAIT2 connections? Maybe You observe same issues as I in Bug 65662
Comment 3 Yann Ylavic 2021-11-22 15:10:00 UTC
(In reply to Lothar from comment #0)
> Also the behavior has changed, with apache 2.4.46 the connection was closed
> smoothly:
> apache -> target ACK
> target -> apache FIN,ACK
> apache -> target ACK
> apache -> target FIN,ACK
> 
> With apache 2.4.51 the last FIN,ACK is not sent anymore but we receive 
> target -> apache RST, ACK after 10 sec.

What happens to the connection on the client side when the target/backend server initiates connection close like the above?
Comment 4 Yann Ylavic 2021-11-22 15:19:40 UTC
Let's track this at a single place (bug 65662) since the reports look very similar.

*** This bug has been marked as a duplicate of bug 65662 ***
Comment 5 Lothar 2021-11-23 08:12:06 UTC
Created attachment 38100 [details]
Trace8 Log filtered by one PID, serving the Request IP-Addresses/hostnames anonymised (

Trace8 Log filtered by one PID, which was serving the Request
IP-Addresses/hostnames anonymised
Comment 6 Lothar 2021-11-23 08:16:32 UTC
(In reply to Piotr Staszeski from comment #2)
> (In reply to Piotr Staszeski from comment #1)
> > pstack 27036
> 
> missclick - @Lothar do You also see related FIN_WAIT2 connections? Maybe You
> observe same issues as I in Bug 65662

The Connection switches immediately from ESTABLISHED into CLOSE_WAIT state
Comment 7 Lothar 2021-11-23 08:19:03 UTC
(In reply to Yann Ylavic from comment #3)
> (In reply to Lothar from comment #0)
> > Also the behavior has changed, with apache 2.4.46 the connection was closed
> > smoothly:
> > apache -> target ACK
> > target -> apache FIN,ACK
> > apache -> target ACK
> > apache -> target FIN,ACK
> > 
> > With apache 2.4.51 the last FIN,ACK is not sent anymore but we receive 
> > target -> apache RST, ACK after 10 sec.
> 
> What happens to the connection on the client side when the target/backend
> server initiates connection close like the above?

The Client receives a 200 OK and everything on client side is OK
Comment 8 Yann Ylavic 2021-11-23 17:28:00 UTC

*** This bug has been marked as a duplicate of bug 65662 ***