Bug 49058 - Prefork MPM tries to accept on closed listener sockets during graceful restarts
Summary: Prefork MPM tries to accept on closed listener sockets during graceful restarts
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_prefork (show other bugs)
Version: 2.2.15
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate
Depends on:
Blocks:
 
Reported: 2010-04-06 19:33 UTC by John Lightsey
Modified: 2018-11-07 21:09 UTC (History)
3 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description John Lightsey 2010-04-06 19:33:47 UTC
This happens intermittently but frequently on my system.  During graceful restarts I see this message in the error log.

[notice] Graceful restart requested, doing restart
[error] (9)Bad file descriptor: apr_socket_accept:
(client socket)

The server restarts properly.

The error message is coming from unixd_accept() and apparently the chain of events that leads to this is....

1. main server gets graceful restart signal.
2. main server signals child processes.
3. main server initiates dummy connections to wake children.
4. child wakes up for the dummy connection 
5. child gets signal for graceful restart and closes all of its listeners.
6. child comes back from apr_pollset_poll with the socket for the dummy connection in pdesc.
7. child calls lr->accept_func() and fails with the error message

The main problem seems to be that right after returning from apr_pollset_poll() in prefork.c, the child process has already set die_now and closed the listeners.

Somewhere between apr_pollset_poll() and "goto got_fd" the child process needs to check die_now so that it doesn't attempt to accept on the closed socket.
Comment 1 John Lightsey 2010-04-06 19:50:18 UTC
Actually, putting the check before "goto got_fd" just tightens the race condition.  unixd_accept() should probably check if the server is shutting down when it fails and not log an error in this case.
Comment 2 Jeff Trawick 2010-04-08 18:52:18 UTC
trunk has code for this situation, though it is only enabled for a certain Fujitsu mainframe platform.  Here is a patch for 2.2.x.  Does this resolve the error message for you?

Index: os/unix/unixd.c
===================================================================
--- os/unix/unixd.c	(revision 923736)
+++ os/unix/unixd.c	(working copy)
@@ -633,6 +633,12 @@
             return APR_EGENERAL;
 #else
         default:
+            /* If the socket has been closed in ap_close_listeners()
+             * by the restart/stop action, we may get EBADF.
+             * Do not print an error in this case.
+             */
+            if (!lr->active && status == EBADF)
+                return status;
             ap_log_error(APLOG_MARK, APLOG_ERR, status, ap_server_conf,
                          "apr_socket_accept: (client socket)");
             return APR_EGENERAL;
Comment 3 John Lightsey 2010-04-08 19:36:37 UTC
Yes, that works perfectly.
Comment 4 Jeff Trawick 2010-04-13 11:39:15 UTC
Here's an updated 2.2.x patch based on what I committed to trunk:

Index: os/unix/unixd.c
===================================================================
--- os/unix/unixd.c	(revision 933664)
+++ os/unix/unixd.c	(working copy)
@@ -633,6 +633,15 @@
             return APR_EGENERAL;
 #else
         default:
+            /* If the socket has been closed in ap_close_listeners()
+             * by the restart/stop action, we may get EBADF.
+             * Do not print an error in this case.
+             */
+            if (!lr->active) {
+                ap_log_error(APLOG_MARK, APLOG_DEBUG, status, ap_server_conf,
+                             "apr_socket_accept failed for inactive listener");
+                return status;
+            }
             ap_log_error(APLOG_MARK, APLOG_ERR, status, ap_server_conf,
                          "apr_socket_accept: (client socket)");
             return APR_EGENERAL;

It can be noisier than the previous patch, as it writes a message at log level debug.
Comment 5 Takashi Sato 2012-02-15 04:20:33 UTC
This issue was resolved as r589761 and r933657 and not backported to 2.2 yet.
Comment 6 William A. Rowe Jr. 2018-11-07 21:09:18 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.