Summary: | regression on httpd reload (could not bind to address 0.0.0.0:80) | ||
---|---|---|---|
Product: | Apache httpd-2 | Reporter: | Thomas Jarosch <thomas.jarosch> |
Component: | mpm_event | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
Status: | CLOSED FIXED | ||
Severity: | normal | CC: | thomas.jarosch |
Priority: | P2 | ||
Version: | 2.4.29 | ||
Target Milestone: | --- | ||
Hardware: | PC | ||
OS: | Linux |
Description
Thomas Jarosch
2017-10-25 15:03:24 UTC
PS: There's no 2.4.29 version in bugzilla yet, 2.4.28 is the highest as of writing. Do you set ListenCoresBucketsRatio? Were you by any chance running tests work prior to 2.4.17 (prior to SO_REUSEPORT)? ListenCoresBucketsRatio is not explicitly set as far as I can tell. Relevant part of httpd.conf: ------------------- Timeout 300 KeepAliveTimeout 300 GracefulShutdownTimeout 1800 <IfModule mpm_event_module> # MinSpareThreads 10 # MaxSpareThreads 250 # # Number of concurrent connections is: ServerLimit * ThreadsPerChild # Result: 160 * 5 -> 800 # StartServers 1 ServerLimit 160 ThreadLimit 5 ThreadsPerChild 5 MaxConnectionsPerChild 5000 </IfModule> Listen 192.168.1.254:80 Listen 192.168.1.254:443 Listen 127.0.0.1:80 Listen 127.0.0.1:443 Listen 447 ------------------- I'll dig through our git history tomorrow at which pointed we switched from httpd 2.2.x to 2.4.x. There was also another switch later on from prefork mpm to the event mpm. 2.4.17 was released around September 2015, we'll see tomorrow.
> Listen 192.168.1.254:80
> Listen 192.168.1.254:443
> Listen 127.0.0.1:80
> Listen 127.0.0.1:443
> Listen 447
> -------------------
Interesting, did you redact? This is not consistent with the failure msg (0.0.0.0:80 / 80)
(In reply to Eric Covener from comment #4) > > Listen 192.168.1.254:80 > > Listen 192.168.1.254:443 > > Listen 127.0.0.1:80 > > Listen 127.0.0.1:443 > > Listen 447 > > ------------------- > > Interesting, did you redact? This is not consistent with the failure msg > (0.0.0.0:80 / 80) sorry, I've grabbed the config from another VM while the original VM was executing a different test. This is how the config looks like: ---- Listen 80 Listen 447 ---- The thread limits etc. are identical. Regarding our upgrade history: We went straight from 2.0.64 to 2.4.7 in January 2014. The switch from the prefork mpm to the event mpm was done already with version 2.4.18 in June 2016. But it looks like I hit the issue last night again while doing a stress test on mod_proxy with the reverted patch. It just doesn't appear every time. I'll now double check the patch was in there. If so, I 'll take version 2.4.27 instead of 2.4.29 to make sure we are not searching for a different issue. For a config like: Listen 127.0.0.1:80 Listen 80 httpd should fail to start by default. This was true up to 2.4.16, but the behaviour accidentally regressed in 2.4.17 with the listener bucket code. From 2.4.26 up it will fail again. It's possible you had a config working with (2.4.17..25) which now fails, but this is not a regression. Alternatively, it's possible you have some unrelated issue. Please provide output from: strace -ebind ./bin/httpd (In reply to Joe Orton from comment #6) > For a config like: > > Listen 127.0.0.1:80 > Listen 80 I don't think we are having a mix of "Listen XX" with an explicit IP address and one without on the same port number. That's a different thing. In the first config I've posted, it's port 443 and *447*. (we use 447 to serve a different SSL cert) All you've shown us is a conflict error message from :80. If there is some issue here other than a config error you need to show full Listen config & strace output I indicated above. (In reply to Joe Orton from comment #8) > All you've shown us is a conflict error message from :80. If there is some > issue here other than a config error you need to show full Listen config & > strace output I indicated above. the "Listen" lines have been provided in comment #5, here they are again: Listen 80 Listen 447 # strace -ebind httpd -f /etc/httpd/conf/httpd.conf bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(4, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 bind(4, {sa_family=AF_INET, sin_port=htons(447), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 The output is from httpd 2.4.29 + revert of the commit mentioned in comment #1. I've been running the mod_proxy tests on two machines in a loop all afternoon. So far no single failure. If I stick to vanilla httpd 2.4.29 or 2.4.27, it fails on "httpd reload". Note: This is no *startup* failure, httpd starts fine. The graceful restart during "httpd reload" -> SIGHUP dies. What's the normal control flow once a SIGHUP is received in the event mpm? Finally tracked it down: The automated system changes the "Listen" directives for a few tests and issues a "httpd reload". While this worked fine in httpd 2.4.25, it stopped working once the SO_REUSEPORT handling was changed in 2.4.26+. I've adapted the tests to issue a "httpd restart" if the Listen directives change and a quick test finished without errors. I'll run the full testsuite over the weekend. Here's a diff of the config file before the "httpd reload": -Listen 172.18.0.1:80 -Listen 172.18.0.1:443 -Listen 192.168.2.1:80 -Listen 192.168.2.1:443 -Listen 127.0.0.1:80 -Listen 127.0.0.1:443 +Listen 80 Listen 447 So Joe was on the right track in comment #6. Changing from "Listen 127.0.0.1:80" to just "Listen 80" is a pretty uncommon (stupid?) thing. For me the matter is resolved, I've adapted the code and all should be fine. If someone else ever stumbles upon this issue, we could think about a warning message in the error log that changing the Listen directives in this way is unsupported during a SIGHUP. My thanks to everyone involved! Have a nice weekend, Thomas (In reply to Thomas Jarosch from comment #10) > So Joe was on the right track in comment #6. Changing from "Listen > 127.0.0.1:80" to just "Listen 80" is a pretty uncommon (stupid?) thing. That's a pretty interesting special case, at least I wasn't aware of it, so thanks a lot for reporting back! I documented this in r1816112 https://httpd.apache.org/docs/trunk/bind.html#reload |