Bug 46671

Summary: Apache periodicaly crashes .
Product: Apache httpd-2 Reporter: Toor <tooriun>
Component: mpm_workerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: REOPENED ---    
Severity: critical CC: chris.tandiono
Priority: P2    
Version: 2.2.11   
Target Milestone: ---   
Hardware: PC   
OS: FreeBSD   
URL: http://no.url
Attachments: Crash core dump

Description Toor 2009-02-07 12:01:05 UTC
FreeBSD 7.0 with PAE.
cimpiled with libpth (with pthread more problems, but i didn't fix it)

After start Apache working up to some hours and got Segmentation fault error in logs.
[Sat Feb 07 19:46:50 2009] [notice] Apache/2.2.11 (Unix) PHP/5.2.8 configured -- resuming normal operations
[Sat Feb 07 20:23:28 2009] [notice] child pid 51670 exit signal Segmentation fault (11)
[Sat Feb 07 20:39:34 2009] [notice] caught SIGTERM, shutting down
[Sat Feb 07 20:40:02 2009] [notice] Apache/2.2.11 (Unix) PHP/5.2.8 configured -- resuming normal operations
[Sat Feb 07 20:41:25 2009] [notice] child pid 55560 exit signal Segmentation fault (11)

After each segfault one of server died and didn't start. Its look in server -status like this:
BEFORE:
R__R_C___R__RR___RR______R_RR_R____W____C___C_____R___________R_
____R___RR__C___C______CCR__R___R___RR____C_R___C______WR___RR_W
_C____C__RR___RCC___R_RR_RR_______R_____R_____CC__RC______C____R
__RRW__RR__C_____R__C_R_R___C_C__R___R_R_W____WR____RC__R_C_C_W_
_W_______R_______RW__RW___C_______RR_R_R____R_R_R___WR_WCC___RR_
_______C__R____R__R______R____R___R________C_RRCC______R__R__R__
C___R_____W____R_W__C_C_____C_RC_______RRR_R_C___RC_R______R__W_
__RW_R_____CW_R_RR_C_R_W_R___R____R____R_R__C___C___W__R________
__C__RR____________R____WC______C____C__R_R_CW______C____R__C__R
RWC__R_______C___________W___C___R____R__R_R_RR____RWC_W__RR__R_

AFTER first sighfault:
R__R_C___R__RR___RR______R_RR_R____W____C___C_____R___________R_
____R___RR__C___C______CCR__R___R___RR____C_R___C______WR___RR_W
_C____C__RR___RCC___R_RR_RR_______R_____R_____CC__RC______C____R
................................................................
__RRW__RR__C_____R__C_R_R___C_C__R___R_R_W____WR____RC__R_C_C_W_
_W_______R_______RW__RW___C_______RR_R_R____R_R_R___WR_WCC___RR_
_______C__R____R__R______R____R___R________C_RRCC______R__R__R__
C___R_____W____R_W__C_C_____C_RC_______RRR_R_C___RC_R______R__W_
__RW_R_____CW_R_RR_C_R_W_R___R____R____R_R__C___C___W__R________
__C__RR____________R____WC______C____C__R_R_CW______C____R__C__R
RWC__R_______C___________W___C___R____R__R_R_RR____RWC_W__RR__R_

AFTER second sighfault:
R__R_C___R__RR___RR______R_RR_R____W____C___C_____R___________R_
____R___RR__C___C______CCR__R___R___RR____C_R___C______WR___RR_W
_C____C__RR___RCC___R_RR_RR_______R_____R_____CC__RC______C____R
................................................................
__RRW__RR__C_____R__C_R_R___C_C__R___R_R_W____WR____RC__R_C_C_W_
_W_______R_______RW__RW___C_______RR_R_R____R_R_R___WR_WCC___RR_
................................................................
_______C__R____R__R______R____R___R________C_RRCC______R__R__R__
C___R_____W____R_W__C_C_____C_RC_______RRR_R_C___RC_R______R__W_
__RW_R_____CW_R_RR_C_R_W_R___R____R____R_R__C___C___W__R________
__C__RR____________R____WC______C____C__R_R_CW______C____R__C__R
RWC__R_______C___________W___C___R____R__R_R_RR____RWC_W__RR__R_


When maximum of allowed server (ServerLimit) is reached (with dead lines)- apache died. For each of segfault I got a apache core.

Core was generated by `httpd'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /usr/local/lib/libaprutil-1.so.3...done.
Loaded symbols for /usr/local/lib/libaprutil-1.so.3
Reading symbols from /usr/local/lib/libexpat.so.1...done.
Loaded symbols for /usr/local/lib/libexpat.so.1
Reading symbols from /usr/local/lib/libapr-1.so.3...done.
Loaded symbols for /usr/local/lib/libapr-1.so.3
Reading symbols from /lib/libcrypt.so.4...done.
Loaded symbols for /lib/libcrypt.so.4
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /usr/local/libexec/apache2/mod_authz_host.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_authz_host.so
Reading symbols from /usr/local/libexec/apache2/mod_env.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_env.so
Reading symbols from /usr/local/libexec/apache2/mod_headers.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_headers.so
Reading symbols from /usr/local/libexec/apache2/mod_setenvif.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_setenvif.so
Reading symbols from /usr/local/libexec/apache2/mod_mime.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_mime.so
Reading symbols from /usr/local/libexec/apache2/mod_status.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_status.so
Reading symbols from /usr/local/libexec/apache2/mod_cgi.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_cgi.so
Reading symbols from /usr/local/libexec/apache2/mod_negotiation.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_negotiation.so
Reading symbols from /usr/local/libexec/apache2/mod_dir.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_dir.so
Reading symbols from /usr/local/libexec/apache2/mod_actions.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_actions.so
Reading symbols from /usr/local/libexec/apache2/mod_alias.so...done.
Loaded symbols for /usr/local/libexec/apache2/mod_alias.so
Reading symbols from /usr/local/libexec/apache2/libphp5.so...done.
Loaded symbols for /usr/local/libexec/apache2/libphp5.so
Reading symbols from /usr/local/lib/mysql/libmysqlclient_r.so.15...done.
Loaded symbols for /usr/local/lib/mysql/libmysqlclient_r.so.15
Reading symbols from /usr/local/lib/libfreetype.so.9...done.
Loaded symbols for /usr/local/lib/libfreetype.so.9
Reading symbols from /usr/local/lib/libpng.so.5...done.
Loaded symbols for /usr/local/lib/libpng.so.5
Reading symbols from /lib/libz.so.4...done.
Loaded symbols for /lib/libz.so.4
Reading symbols from /usr/local/lib/libjpeg.so.9...done.
Loaded symbols for /usr/local/lib/libjpeg.so.9
Reading symbols from /usr/local/lib/libcurl.so.4...done.
Loaded symbols for /usr/local/lib/libcurl.so.4
Reading symbols from /usr/lib/libssl.so.5...done.
Loaded symbols for /usr/lib/libssl.so.5
Reading symbols from /lib/libcrypto.so.5...done.
Loaded symbols for /lib/libcrypto.so.5
Reading symbols from /usr/local/lib/libxml2.so.5...done.
Loaded symbols for /usr/local/lib/libxml2.so.5
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /usr/local/lib/php/extensions/no-debug-zts-20060613/eaccelerator.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-zts-20060613/eaccelerator.so
Reading symbols from /usr/local/lib/php/extensions/no-debug-zts-20060613/imagick.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-zts-20060613/imagick.so
Reading symbols from /usr/local/lib/libMagickWand.so.2...done.
Loaded symbols for /usr/local/lib/libMagickWand.so.2
Reading symbols from /usr/local/lib/libMagickCore.so.2...done.
Loaded symbols for /usr/local/lib/libMagickCore.so.2
Reading symbols from /usr/lib/libbz2.so.3...done.
Loaded symbols for /usr/lib/libbz2.so.3
Reading symbols from /usr/lib/libgomp.so.1...done.
Loaded symbols for /usr/lib/libgomp.so.1
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x28163188 in sem_init () from /lib/libthr.so.3
[New Thread 0x28301100 (LWP 100564)]
(gdb) thread apply all bt full

Thread 1 (Thread 0x28301100 (LWP 100564)):
#0  0x28163188 in sem_init () from /lib/libthr.so.3
No symbol table info available.
#1  0x282fc5b9 in GOMP_ordered_start () from /usr/lib/libgomp.so.1
No symbol table info available.
#2  0x282fd145 in omp_get_nested () from /usr/lib/libgomp.so.1
No symbol table info available.
#3  0x280c63f4 in ?? () from /libexec/ld-elf.so.1
No symbol table info available.
#4  0x280cefc0 in ?? ()
No symbol table info available.
#5  0xbf7fe9f8 in ?? ()
No symbol table info available.
#6  0x282f9bc1 in ?? () from /usr/lib/libgomp.so.1
No symbol table info available.
#7  0x280c6560 in ?? () from /libexec/ld-elf.so.1
No symbol table info available.
#8  0x280c63f4 in ?? () from /libexec/ld-elf.so.1
No symbol table info available.
#9  0xbf7fe9f8 in ?? ()
No symbol table info available.
#10 0x280a43bf in dlsym () from /libexec/ld-elf.so.1
No symbol table info available.

So, I tried to change AcceptMutex, change "worker" options, but it doesn't help
My option is now:
ServerLimit 64
StartServers 10
MaxClients 1024
MinSpareThreads 75
MaxSpareThreads 1024

ThreadsPerChild 64
#ThreadLimit 100

MaxRequestsPerChild 0

KeepAlive Off
MaxKeepAliveRequests 600
KeepAliveTimeout 3
Comment 1 Toor 2009-02-09 10:48:43 UTC
*** Bug 46684 has been marked as a duplicate of this bug. ***
Comment 2 Toor 2009-02-09 10:49:45 UTC
Created attachment 23242 [details]
Crash core dump
Comment 3 Nick Kew 2009-05-25 15:10:57 UTC
You are using a threaded MPM not only with mod_php, but with a number of libraries known not to be thread-safe.  When you do that, occasional segfaults are to be expected.
Comment 4 Toor 2009-06-06 02:14:04 UTC
(In reply to comment #3)
> You are using a threaded MPM not only with mod_php, but with a number of
> libraries known not to be thread-safe.  When you do that, occasional segfaults
> are to be expected.
It's possible, but I talk not about segfault, but about behaviour after segfault. As described in doc, apache child must restart (or just die) after segfault, but his not. So, after this happend many times apache have no more limit for run new childs and crashes as result of limit excedeed (not because segfault happend).
Comment 5 Chris Tandiono 2011-07-03 18:07:35 UTC
I'm getting the exact same error with almost the exact same backtrace, but I'm using prefork MPM. This doesn't seem possible. This is FreeBSD 8.1.