Bug 54989

Summary: Apache Segfaults on OS X at random - how can I diagnose?
Product: Apache httpd-2 Reporter: Ryan Barclay <ryan>
Component: CoreAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: normal Keywords: MassUpdate
Priority: P2    
Version: 2.2.22   
Target Milestone: ---   
Hardware: Macintosh   
OS: All   

Description Ryan Barclay 2013-05-17 22:03:48 UTC
We have been having intermittent Apache crashes on OS X 10.7.5 (Lion Server).  These crashes seem to happen at random and we've tried everything to find the cause.

We started noticing them more when we changed was MaxRequestsPerChild to 10 to stop memleaks on some scripts we are running.  We noticed that the crashes seem to happen when the children are coming to the end of their life and have served their 10 requests.

We also notice the segfaults come in waves.  For example, we won't notice any for three hours and the all of a sudden they happen every minute for 10 faults and then go quiet again (possible somebody running a vulnerability scanner).
 
The problem may be with PHP or Apache itself.  We have also noticed the same crashes with XCache/APC disabled.
 
The crashes seem to always be on "Thread 1" which is kernel/OS side of things most of the time - e.g.:

    Thread 1 Crashed:
    0   libsystem_kernel.dylib                  0x00007fff910eb192 __workq_kernreturn + 10
    1   libsystem_c.dylib                       0x00007fff8fe96594 _pthread_wqthread + 758
    2   libsystem_c.dylib                       0x00007fff8fe97b85 start_wqthread + 13
 
Any help, suggestions, tips or bug fixed are very greatly appreciated.  Although, please do not suggest upgrading PHP/Apache or OS X - we need a fix for the system in its current state; we're not in a position to upgrade/recompile anything on such a live server so feel free to let us know of patches or fixes (band-aids!) either on the OS side or Apache side.

We would love help in finding the cause or what page request creates the crash (however we can't use mod_whatkilledus because we haven't compiled with exception hooks.

Any suggested Apache modules and debugging tools are greatly appreciated.
 
Our crash report:

<pre>
Process:         httpd [52830]
Path:            /usr/sbin/httpd
Identifier:      httpd
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  httpd [86996]
 
 
Date/Time:       2013-05-15 09:39:36.067 +0100
OS Version:      Mac OS X Server 10.7.5 (11G63)
Report Version:  9
 
 
Crashed Thread:  1  Dispatch queue: com.apple.libdispatch-manager
 
 
Exception Type:  EXC_CRASH (SIGSEGV)
Exception Codes: 0x0000000000000000, 0x0000000000000000
 
 
Application Specific Information:
objc[86996]: garbage collection is OFF
 
 
Thread 0:: Dispatch queue: com.apple.main-thread
0   libphp5.so                              0x0000000103dbd7e2 _zend_hash_add_or_update + 75
1   libphp5.so                              0x0000000103d6f5ad zend_symtable_update + 208
2   libphp5.so                              0x0000000103d7047b php_register_variable_ex + 1584
3   libphp5.so                              0x0000000103b5311d php_sapi_filter + 964
4   libphp5.so                              0x0000000103d6a462 sapi_getenv + 99
5   libphp5.so                              0x0000000103d74a7d php_security_log + 193
6   libphp5.so                              0x0000000103dcccd2 _zend_mm_free_canary_int + 110
7   libphp5.so                              0x0000000103d9786e _efree + 52
8   libphp5.so                              0x0000000103db07b8 _zval_dtor_func + 56
9   libphp5.so                              0x0000000103daa6ce destroy_op_array + 181
10  libphp5.so                              0x0000000103dbc712 zend_hash_destroy + 53
11  libphp5.so                              0x0000000103daa849 destroy_zend_class + 100
12  libphp5.so                              0x0000000103dbc464 zend_hash_apply_deleter + 189
13  libphp5.so                              0x0000000103dbc50f zend_hash_reverse_apply + 102
14  libphp5.so                              0x0000000103da8dd2 shutdown_executor + 650
15  libphp5.so                              0x0000000103db1963 zend_deactivate + 99
16  libphp5.so                              0x0000000103d61a4d php_request_shutdown + 583
17  libphp5.so                              0x0000000103e2a5d4 php_handler + 1186
18  httpd                                   0x0000000103755031 ap_run_handler + 97
19  httpd                                   0x0000000103755bd6 ap_invoke_handler + 422
20  httpd                                   0x0000000103791616 ap_internal_redirect + 118
21  mod_rewrite.so                          0x0000000103a5dc70 handler_redirect + 256
22  httpd                                   0x0000000103755031 ap_run_handler + 97
23  httpd                                   0x0000000103755bd6 ap_invoke_handler + 422
24  httpd                                   0x00000001037909cd ap_process_request + 125
25  httpd                                   0x000000010378c5c3 ap_process_http_connection + 131
26  httpd                                   0x000000010376c651 ap_run_process_connection + 97
27  httpd                                   0x000000010376cc31 ap_process_connection + 113
28  httpd                                   0x0000000103799c73 child_main + 1699
29  httpd                                   0x0000000103799eb3 make_child + 435
30  httpd                                   0x000000010379a23f perform_idle_server_maintenance + 703
31  httpd                                   0x000000010379a96c ap_mpm_run + 1740
32  httpd                                   0x0000000103760077 main + 4055
33  httpd                                   0x0000000103754374 start + 52
 
 
Thread 1 Crashed:: Dispatch queue: com.apple.libdispatch-manager
0   libsystem_kernel.dylib                  0x00007fff8e2557e6 kevent + 10
1   libdispatch.dylib                       0x00007fff89870786 _dispatch_mgr_invoke + 923
2   libdispatch.dylib                       0x00007fff8986f316 _dispatch_mgr_thread + 54
 
 
Thread 1 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000004  rbx: 0x00007fff763a31a0  rcx: 0x00000001432cb528  rdx: 0x0000000000000000
  rdi: 0x000000000000001b  rsi: 0x0000000000000000  rbp: 0x00000001432cb6a0  rsp: 0x00000001432cb528
   r8: 0x0000000000000001   r9: 0x0000000000000000  r10: 0x00000001432cb538  r11: 0xffffff80002dad60
  r12: 0x00007fff763a2540  r13: 0x0000000000000000  r14: 0x00007fff763a3230  r15: 0x0000000000000000
  rip: 0x00007fff8e2557e6  rfl: 0x0000000000000247  cr2: 0x00007fff763a4310
Logical CPU: 0

 
External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 4267
    thread_create: 0
    thread_set_state: 0
 
 
VM Region Summary:
ReadOnly portion of Libraries: Total=143.3M resident=54.1M(38%) swapped_out_or_unallocated=89.2M(62%)
Writable regions: Total=96.9M written=30.2M(31%) resident=36.1M(37%) swapped_out=0K(0%) unallocated=60.9M(63%)
 
REGION TYPE                      VIRTUAL
===========                      =======
MALLOC                             86.3M
MALLOC guard page                    48K
STACK GUARD                        56.0M
Stack                              8712K
VM_ALLOCATE                         688K
__CI_BITMAP                          80K
__DATA                             10.9M
__LINKEDIT                         51.0M
__TEXT                             92.3M
__UNICODE                           544K
mapped file                         1.0G
shared memory                        12K
===========                      =======
TOTAL                               1.3G
</pre>
Comment 1 William A. Rowe Jr. 2018-11-07 21:09:04 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.