Bug 22516

Summary: Apache 2 crashes with Acept Mutex error when stressed with 20 users on HP-UX 11.00
Product: Apache httpd-2 Reporter: Arthur Luimes <arthur.luimes>
Component: CoreAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: CLOSED DUPLICATE    
Severity: blocker    
Priority: P3    
Version: 2.0.47   
Target Milestone: ---   
Hardware: HP   
OS: HP-UX   

Description Arthur Luimes 2003-08-18 14:17:25 UTC
During a loadtest with 20 concurrent users, Apache 2.0.47 (using all default 
config values) crashes on HP-UX 11.00. The error messages are:
[Mon Aug 18 15:35:06 2003] [emerg] (28)No space left on device: couldn't grab 
the accept mutex
[Mon Aug 18 15:35:06 2003] [alert] Child 2349 returned a Fatal error...
Apache is exiting!
[Mon Aug 18 15:35:06 2003] [emerg] (36)Identifier removed: couldn't grab the 
accept mutex
[Mon Aug 18 15:35:06 2003] [emerg] (36)Identifier removed: couldn't grab the 
accept mutex
[Mon Aug 18 15:35:06 2003] [emerg] (36)Identifier removed: couldn't grab the 
accept mutex
... etc....

There is plenty of space on all devices: this could not be the issue. I've 
tried adjusting some HP paramateres as in the HP thread:
http://forums.itrc.hp.com/cm/QuestionAnswer/0,,0xf91e36e69499d611abdb0090277a778
c,00.html

All to no avail. 

My system is now an easy target for DOS attacks. What to do?

P.S. I've load-tested the sever with 10 concurrent users and find no problems.
Comment 1 Jeff Trawick 2003-08-18 14:27:58 UTC
What kind of accept mutex are you using?

To determine this:

1st see if there is an AcceptMutex directive in your configuration file
if not, the output of "apachectl -V" tells what is the compiled-in default

I think it will say

 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE

which means that SysV semaphores are used for cross-process locks
and pthread mutexes are used for intra-process locks

meanwhile, ENOSPC (no space left on device) is an error associated with SysV
semaphores, and it has nothing to do with available filesystem space

Assuming you're really using SysV semaphores:

a) the problem may be some OS tuning issue...  increase any system tuning
associated with SysV sems until the problem goes away
b) if you can't get it tuned properly, switch to a different mutex mechanism,
such as fcntl, by putting something like this in your config file:

  AcceptMutex fcntl

Note that fcntl mutexes have their own issues.  For starters, if your server
root is on a remote filesystem, you'll need to use the LockFile directive.
Comment 2 Arthur Luimes 2003-08-18 15:24:13 UTC
Jeff,

The setting AcceptMutex fcntl solved the issue! Thanks a lot!!

What kind of issues you mentioned below can I expect with fcntl?
Comment 3 Jeff Trawick 2003-08-19 11:20:37 UTC
What issues to expect with fcntl?  I have relatively little HP-UX specific
knowledge.  Some of the issues that have come up on other platforms include:

fcntl may not perform as well as SysV semaphore or cross-process pthread locks

fcntl may have tuning requirements also, and if you add additional child
processes to the server to handle increased load you may suddenly hit an
fcntl-related tuning issue

BTW, somebody reported the same problem a couple of days earlier (PR 22484). 
I'll change the closing status of this one to a dup of that one.
Comment 4 Jeff Trawick 2003-08-19 11:20:51 UTC

*** This bug has been marked as a duplicate of 22484 ***