Bug 34508 - Randomly slow graceful restart, not accepting new connections.
Summary: Randomly slow graceful restart, not accepting new connections.
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_prefork (show other bugs)
Version: 2.0-HEAD
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate
Depends on:
Blocks:
 
Reported: 2005-04-19 00:12 UTC by Carlos Gonz
Modified: 2018-11-07 21:09 UTC (History)
1 user (show)



Attachments
Debug Files (563.71 KB, application/zip)
2013-12-26 11:11 UTC, Jerald Johnson
Details
Debug Files (563.71 KB, application/zip)
2013-12-26 11:12 UTC, Jerald Johnson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Carlos Gonz 2005-04-19 00:12:39 UTC
This is observed in 2.0.53 and 2.0.54 versions, prefork operated. The server use
is medium (typically 100 to 200 simultaneous connections), no PHP, no mod_perl,
only cgi.

Every hour, I move (mv) the log to a new filename, send a graceful restart to
ensure using a new one, and wait 60 seconds (other values tested without
changes) to copy it to another host to process it (scp). Some minutes later, the
old log is deleted.

I see the server stops to accept new connections just after the graceful restart
is requested, bandwidth measurements shows a clear down graphic, and time
required to accept new ones vary from a few seconds (97% cases) to a few
minutes, keeping the site unnecessarilly down. I expect graceful restart don't
causes to stop accepting new connections, as I understand this is the normal
SIGUSR1 action required.

 A test adding a sleep of five seconds after moving log and before requesting
the graceful restart causes no effect.

These are the compiling options used:

CC="gcc" CFLAGS="-O2 -fomit-frame-pointer -fno-strict-aliasing -fno-common -pipe
-mpreferred-stack-boundary=2 -march=i686" ./configure
--prefix=/usr/local/apache/apache_1 --enable-modules="deflate headers info cgi
speling rewrite" --disable-so --disable-suexec --disable-cgid --disable-expires
--disable-logio --disable-imap --disable-negotiation --disable-autoindex
--disable-include --disable-userdir

There are two more Apache2 servers on the same host (using different IP/port
combinations, of course, and different PID files), one used for SSL connections
and another one to serve PHP and mod_perl scripts, these others have no log
processing, and no graceful restarts are requested for them.

May be interesting to show this configuration parts:

<IfModule prefork.c>
StartServers        64
MinSpareServers     32
MaxSpareServers     64
MaxClients         250
MaxRequestsPerChild  0
</IfModule>
Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 10

There is almost no swap, I don't see LoadAverage or CPU% too high (<1.00, <30%),
but is difficult to catch it just at the exact 'failing time'. There is no local
databases, SMTP servers or other functionalities wich suspected to cause load
for this host (P4 2,8 Ghz, 1GB RAM), is maintained only to host three Apache2
servers.

Also note that if I use a restart instead of a graceful restart, the results are
just the same: too much time to kill the server -not allways, only about 3%-,
from 30 seconds to 8 minutes, and then the new connections accepted when all
finished and restarted.

Too much time to shutdown childs and not accepting new connections when issuing
a graceful restart is the problem I see.

Thank to all the readers.
Comment 1 Joe Orton 2005-09-08 03:38:08 UTC
What's written to the error log?  How are you initiating the restart, using
"apachectl graceful"?
Comment 2 Carlos Gonz 2005-09-08 11:09:53 UTC
Now I'm using rotatelogs to bypass the problem, and fortunatelly it takes less 
load to the server than I expect.

The command used for graceful restart was 'apachectl graceful'.

The server logs don't show any abnormal messages, I only see the normal 
restart messages (I can't see it directly now, sorry) at the real restart 
time, wich is some times a few minutes after the graceful command is issued. 
Prior to the restart messages, there aren't any suspicious message logged, 
only a unavailability to connect to it.
Comment 3 Joe Orton 2005-09-25 23:19:26 UTC
This lacks enough information to reproduce; we'd need, for example, the output
of running strace against the parent process to diagnose further.
Comment 4 Arkadiusz Miskiewicz 2013-04-29 06:57:47 UTC
Looks like #54852
Comment 5 Jerald Johnson 2013-12-26 10:34:47 UTC
I will work on getting you an strace, along with other information.
Comment 6 Jerald Johnson 2013-12-26 11:11:37 UTC
Created attachment 31158 [details]
Debug Files
Comment 7 Jerald Johnson 2013-12-26 11:12:05 UTC
Created attachment 31159 [details]
Debug Files

Here is a strace of the graceful issue, a count of defunct proccesses, a full `ps auxf', httpd -l, httpd -L, httpd -V, and httpd -v
Comment 8 William A. Rowe Jr. 2018-11-07 21:09:27 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.