Hi all,

I've been working with a couple PHP maintainers (Niels Dossche and Derick Rethans) to troubleshoot some segmentation faults that occur on line 597 of sapi/apache2handler/sapi_apache2.c in PHP 8.1.16. The lengthy discussion and troubleshooting can be found here: https://github.com/php/php-src/issues/10737

After much testing, I've discovered that beginning with Apache version 2.4.50, PHP begins to produce segmentation faults under highly concurrent traffic loads. The segmentation faults do not occur with Apache version 2.4.49 and earlier. I've only been able to generate these segmentation faults by introducing a significant level of production traffic to the web server.

The PHP version does not seem to matter. I tested against PHP 7.3.27, 8.1.16 and 8.2.3 -- all of them produce segmentation faults on the same line of code. I've also confirmed that the segmentation faults persist up through version 2.4.54 of Apache.

Here is the GDB output for one of those segmentation faults (in this example I was running Apache 2.4.54 with PHP 7.3.27): https://pastebin.com/raw/4fykKB8t  -- that output contains 'bt', 'bt full', 'layout asm' and 'info registers' output.

For your convenience, here's the 'bt' output:

(gdb) bt
#0  0x00007f5a763dfa67 in php_handler (r=0x7f5958006d30) at /usr/local/src/php-7.3.27/sapi/apache2handler/sapi_apache2.c:583
#1  0x0000000000455830 in ap_run_handler (r=r@entry=0x7f5958006d30) at config.c:169
#2  0x0000000000455dc6 in ap_invoke_handler (r=r@entry=0x7f5958006d30) at config.c:443
#3  0x000000000046a87b in ap_process_async_request (r=0x7f5958006d30) at http_request.c:452
#4  0x000000000046a9ce in ap_process_request (r=r@entry=0x7f5958006d30) at http_request.c:487
#5  0x00007f5a78502d76 in h2_task_process_request (c=0x7f5958000d00, task=<optimized out>) at h2_task.c:671
#6  h2_task_process_conn (c=0x7f5958000d00) at h2_task.c:713
#7  h2_task_process_conn (c=0x7f5958000d00) at h2_task.c:700
#8  0x000000000045e670 in ap_run_process_connection (c=c@entry=0x7f5958000d00) at connection.c:42
#9  0x00007f5a78504007 in h2_task_do (task=0x7f5958004d20, thread=thread@entry=0x1a20e28, worker_id=<optimized out>) at h2_task.c:631
#10 0x00007f5a78507ad0 in slot_run (thread=0x1a20e28, wctx=0x1a32310) at h2_workers.c:263
#11 0x00007f5a7d3821ca in start_thread () from /lib64/libpthread.so.0
#12 0x00007f5a7cdeae73 in clone () from /lib64/libc.so.6
(gdb)

I don't want to speak for the PHP maintainers (their expertise far exceeds mine), but if you review our discussion located here https://github.com/php/php-src/issues/10737 , you'll find that Niels has identified that the problem is due to the 'TSRMLS_CACHE' being null in PHP's 'TSRM/TSRM.c', but it's unclear how that's possible to have occurred inside the handler itself ('sapi/apache2handler/sapi_apache2.c'). It's also not clear if this is an Apache fix, a PHP fix or some combination thereof.

Here's the system and Apache version output from my most recent test environment, but as I mentioned before, these segmentation faults occur starting with version 2.4.50 and persist through the latest version.

[root@dev ~]# cat /etc/*release*
CentOS Stream release 8
cat: /etc/lsb-release.d: Is a directory
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 8
CentOS Stream release 8
cpe:/o:centos:centos:8
[root@dev ~]# uname -a
Linux dev.******.com 4.18.0-448.el8.x86_64 #1 SMP Wed Jan 18 15:02:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@dev ~]# /usr/local/apache/bin/httpd -V
Server version: Apache/2.4.54 (Unix)
Server built:   Mar  6 2023 11:55:44
Server's Module Magic Number: 20120211:124
Server loaded:  APR 1.7.0, APR-UTIL 1.6.1, PCRE 10.32 2018-09-10
Compiled using: APR 1.7.0, APR-UTIL 1.6.1, PCRE 10.32 2018-09-10
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_PIDLOG="/var/logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
[root@dev ~]# /usr/local/apache/bin/httpd -l
Compiled in modules:
  core.c
  mod_so.c
  http_core.c
  event.c
[root@dev ~]#

Please let me know if you require any additional debug information. Your assistance in tracking down these segmentation faults would be highly appreciated!

- Elliot