Bug 58550 - During load, Apache 2.2.x with prefork MPM didn’t accept the incoming connections in reasonable time
Summary: During load, Apache 2.2.x with prefork MPM didn’t accept the incoming connect...
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_prefork (show other bugs)
Version: 2.2-HEAD
Hardware: All All
: P2 enhancement (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate
Depends on:
Blocks:
 
Reported: 2015-10-26 11:25 UTC by horowity@checkpoint.com
Modified: 2018-11-07 21:10 UTC (History)
0 users



Attachments
Patch to allow multiple accept mutexes (5.06 KB, patch)
2015-10-26 11:25 UTC, horowity@checkpoint.com
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description horowity@checkpoint.com 2015-10-26 11:25:46 UTC
Created attachment 33222 [details]
Patch to allow multiple accept mutexes

Hello Apache gurus.

I was working on a project which used Apache 2.2.x with prefork MPM (using flock as mutex method) on Linux machine (with 20 cores), and run into the following problem.

During load, when number of Apache child processes get beyond some point (~3000 processes) - Apache didn’t accept the incoming connections in reasonable time (seen in netstat as SYN_RECV).

I found a document about Apache Performance Tuning [1], in which there is an idea to improve the performance by:

“Another solution that has been considered but never implemented is to partially serialize the loop -- that is, let in a certain number of processes. This would only be of interest on multiprocessor boxes where it's possible that multiple children could run simultaneously, and the serialization actually doesn't take advantage of the full bandwidth. This is a possible area of future investigation, but priority remains low because highly parallel web servers are not the norm.”

I wrote a small patch (aligned to 2.2.31) that implements this idea – create 4 mutexes and spread the child processes across the mutexes (by getpid() % mutex_number).

So in any given time - 4 ideal child processes are expected [2] to wait in the “select loop”.

Once a new connection arrive - 4 processes are awake by the OS: 1 will succeed to accept the socket (and will release his mutex) and 3 will return to the “select loop”.

This solved my specific problem and allowed me to get more load on the machine.

My questions to this forum are:

1.       Do you think this is a good implementation of the suggested idea?

2.       Any pitfalls I missed?

3.       Would you consider accepting this patch to the project?
If so, could you guide me what else needs to be done for acceptances?
I know there is a need for configuration & documentation work - I’ll work on once the patch will be approved…

4.       Do you think ‘4’ is a good default for the mutexes number? What should be the considerations to set the default?

5.       Does such implementation relevant for other MPMs (worker/event)?

Any other feedback is welcome.

[1] http://httpd.apache.org/docs/2.2/misc/perf-tuning.html, accept Serialization - Multiple Sockets section.

[2] There is no guarantee that exactly 4 processes will wait as all processes of “getpid() % mutex_number == 0” might be busy in a given time. But this sounds to me like a fair limitation.

Note: flock give me the best results, still it seems to be with n^2 complexity (where ‘n’ is the number of waiting processes), so reducing the number of processes waiting on each mutex give exponential improvement.

Regards,

Yehezkel Horowitz
Check Point Software Technologies Ltd.
Comment 1 William A. Rowe Jr. 2018-11-07 21:10:01 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.