Bug 63933 - CLOSE_WAITs using Apache httpd-2.4.27.1-64 mpm_worker
Summary: CLOSE_WAITs using Apache httpd-2.4.27.1-64 mpm_worker
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.27
Hardware: Other Linux
: P2 major (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-19 04:02 UTC by Santosh
Modified: 2019-11-19 04:04 UTC (History)
1 user (show)


Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Santosh 2019-11-19 04:02:48 UTC 
Hi Teaam,

We are using Apache httpd-2.4.27.1-64 as a reverse proxy on top of multiple tomcat web applications servers, have a F5 Load Balancer in front of Apache.

Once in a while there are many sockets in CLOSE_WAITs(anywhere between 100 and 400, worst I have seen is >2000) that do not clear for more than 10-15 mins causing an outage to the web applications.
During such a situation the RAM on the RHEL machine where Apache web server is being run is > 100% committed.

I was using Apache httpd-2.2.31.1-64 earlier and upgraded to httpd-2.4.27.1-64 but the problem persists.

The number of worker processes is >30 usually when there are too many CLOSE_WAITs.

Some key setting are as follows:
Timeout 300
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 5

Worker Module settings are as follows:
<IfModule mpm_worker_module>
    ServerLimit         128
    StartServers         10
    MaxClients         4096
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      32
    MaxRequestsPerChild   0
    ListenBackLog      2048
</IfModule>

Apache is running on RHEL 2.6.32-754.23.1.el6.x86_64.

# ulimit -a -S
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 160660
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 160660
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
# ulimit -a -H
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 160660
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 160660
virtual memory          (kbytes, -v) unlimited

Please see if there are any related bugs or any notes that I can refer that can help solve the issue.

Thanks,
Santosh