Bug 48471

Summary: Apache Restarts Randomly When Using Worker
Product: Apache httpd-2 Reporter: Hay <theking>
Component: mpm_workerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Severity: major    
Priority: P2    
Version: 2.2.14   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description Hay 2010-01-02 02:56:32 UTC
From time to time, Apache keeps restarting (sometimes rarely, sometimes quite often).

grep -R "Doing graceful restart" /etc/httpd/logs/error_log | awk 'BEGIN {FS="]"};{print $1}'| cut -d "[" -f 2

Tue Dec 29 16:11:06 2009
Tue Dec 29 16:24:40 2009
Tue Dec 29 16:25:11 2009
Tue Dec 29 16:25:12 2009
Tue Dec 29 16:28:44 2009
Tue Dec 29 16:37:29 2009
Tue Dec 29 16:46:17 2009
Tue Dec 29 16:55:13 2009

I turned on the LogLevel from warn to debug and Apache and uncovered the following message in the logs:

[Tue Dec 29 04:35:05 2009] [notice] SIGUSR1 received. Doing graceful restart [Tue Dec 29 04:35:05 2009] [debug] worker.c(791): (22)Invalid argument:
apr_proc_mutex_unlock failed. Attempting to shutdown process gracefully.

Switching to prework solved the issue. Recompiled many times, still same issue.

CentOS 5.4 x64
Comment 1 Hay 2010-01-02 03:01:01 UTC
PHP 5.2.12
Comment 2 Nick Kew 2010-01-02 03:58:06 UTC
It's well-known and well-documented, you don't use worker (or other threaded MPM) with non-thread-safe software.

That's why all the distros give you prefork if you want mod_php.

If you want worker+php, run the php out-of-process, e.g. with CGI or fastcgi.
Comment 3 Hay 2010-01-02 04:08:52 UTC
I am running suPHP + fCGI
Comment 4 Hay 2010-01-02 05:46:26 UTC
I am not using mod_php, I am running on suPHP + fCGI
Comment 5 Jeff Trawick 2010-01-02 05:52:03 UTC
> [notice] SIGUSR1 received. Doing graceful restart

Some process is sending the SIGUSR1 signal to the httpd parent process; this should be a separate httpd process, when invoked via "apachectl graceful" or "httpd -k graceful".

If it isn't a script or user explicitly triggering graceful restart, I don't know a straightforward way to capture the sender.

Perhaps a script that watches the log file for that message could capture the list of active processes?  (Hopefully someone has a better idea.)

>debug] worker.c(791): (22)Invalid argument: apr_proc_mutex_unlock failed. Attempting to shutdown process gracefully.

This is an uninteresting error which occurs during graceful restart processing and can be ignored.
Comment 6 Hay 2010-01-06 00:28:37 UTC
Well whatever process is making these graceful restarts, is not using /etc/init.d/httpd. I see restarts at 1900, but nothing has touched that init script since I have last. I also checked and a /scripts/restartsrv_httpd doesn't issue a graceful restart either , just a regular restart.
Comment 7 Hay 2010-01-06 09:35:12 UTC
From the logs, I believe I might have finally found something interesting from the script:

[root@host] ~ >> grep -R safeaprestart /root/restartWatch-*

/root/restartWatch-01:16::01-06-2010:root 24474 0.0 0.1 28456 4040 ? SN 01:16
0:00 safeaprestart - doing restart /root/restartWatch-09:30::01-06-2010:root
10721 2.5 0.1 28456 4044 ? RN 09:30 0:00 safeaprestart - doing restart /root/restartWatch-09:47::01-06-2010:root 25993 1.0 0.1 28452 4040 ? SN 09:47
0:00 safeaprestart - doing restart /root/restartWatch-10:05::01-06-2010:root
21235 0.0 0.1 28452 4040 ? SN 10:05 0:00 safeaprestart - doing restart /root/restartWatch-12:05::01-06-2010:root 1042 0.0 0.1 28456 4036 ? SN 12:05
0:00 safeaprestart - doing restart

As you can see, when these restarts happen, there is a mysterious "safeaprestart" process running. A little research seems to indicate that is a component of tailwatchd. Also a little research reveals that everytime you add a account, Apache is restarted with that process as well:

$output .= Whostmgr::UI::setstatus("Restarting apache"); Cpanel::HttpUtils::safeaprestart();
$output .= Whostmgr::UI::setstatusdone();

For now I have disabled monitoring of Apache from tailwatchd. Now if the issue was purely tailwatchd restarting Apache even though it didn't need to be, this won't cause any downtime. On the other hand if Apache is really failing due to some other issue (and it doesn't appear to be) it will continue to fail, but won't be automatically restarted by tailwatchd.

As of now, tailwatchd monitoring of Apache is disabled. So if the restarts cease to happen now, we will know tailwatchd was the culprit.
Comment 8 Ruediger Pluem 2010-01-06 09:44:41 UTC
3rd party issue, non httpd related.
Comment 9 William A. Rowe Jr. 2010-01-11 20:47:25 UTC
Huh niq?

Nothing in PHP [itself] is a problem - in fact it works very well in any
number of distributions.  Combining PHP with extensions that access non-threadsafe
libraries is - obviously - a problem.

But please leave this fud on PHP's documentation commentary sections, where the
misinformed, blanket commentary belongs.

As noted by rplume, this is not the place for such problem reports.