Bug 62697 - `ExtendedStatus Off` directive when using mod_systemd causes systemctl to hang
Summary: `ExtendedStatus Off` directive when using mod_systemd causes systemctl to hang
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Other Modules (show other bugs)
Version: 2.5-HEAD
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Depends on:
Reported: 2018-09-07 20:20 UTC by Doug Knight
Modified: 2018-09-11 12:53 UTC (History)
0 users

(UNTESTED) Proposed Resolution (644 bytes, patch)
2018-09-07 20:41 UTC, Doug Knight
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Doug Knight 2018-09-07 20:20:17 UTC
When using mod_systemd on a system with a systemd httpd service of type notify, we have encountered a scenario where the command `systemctl reload httpd` never returns.  In this case, systemd continues to report the service as reloading, even though httpd respawned all of its server processes.

I have tracked the problem down to the way in which an `ExtendedStatus Off` directive in the local configuration is interacting with the code introduced here: http://svn.apache.org/viewvc?view=revision&revision=1802251 .  The general sequence of events is:

1. User runs `systemctl reload httpd`.
2. systemd forks off an `httpd -k graceful` process, which exits successfully.
3. mod_systemd's systemd_pre_config hook sends a notification to systemd that httpd is reloading, then sets ap_extended_status.
4. While reloading its configuration, httpd responds to an `ExtendedStatus Off` directive by clearing ap_extended_status.
5. Subsequent calls to mod_systemd's systemd_monitor hook short-circuit when they see that ap_extended_status is cleared so httpd stops sending status updates to systemd.
6. systemd indefinitely reports httpd's state as reloading because it never receives a READY=1 notification from mod_systemd's systemd_monitor hook.
7. The `systemctl reload httpd` command never exits.

I would expect httpd tell systemd when it finished reloading its configuration regardless of the state of ap_extended_status.  I believe sending a notification such as "READY=1\nSTATUS=ExtendedStatus is disabled." before short-circuiting when ap_extended_status is cleared should resolve the issue, but I haven't had an opportunity to setup a build environment to confirm.
Comment 1 Doug Knight 2018-09-07 20:41:03 UTC
Created attachment 36140 [details]
(UNTESTED) Proposed Resolution

I am attaching a patch illustrating what I believe would address this issue.
Comment 2 Doug Knight 2018-09-07 22:11:40 UTC
I have tested the previously attached patch by rebuilding the httpd package in Amazon Linux 2 (httpd-2.4.34-1.amzn2.1.0 including vendor patches), and confirmed that `systemctl reload httpd` returns properly after the patch is applied in that environment.  I still do not have a pristine build environment.
Comment 3 Joe Orton 2018-09-11 12:45:33 UTC
Good catch, thanks for the report and patch.  I think it would be slightly better to do the READY=1 in a post_config hook actually outside of the monitor hook.  At this point the serer is listening TCP ports so should be acceptable to say the service is "started".  (There is a slight trade-off here since we can't guarantee some failure after saying READY=1 before entering the MPM, but that could happen anyway.)
Comment 4 Joe Orton 2018-09-11 12:53:57 UTC
Committed in r1840554, please test it if you can!