Bug 61237 - apache2 2.4.26 segfaults when http2 module is enabled
Summary: apache2 2.4.26 segfaults when http2 module is enabled
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_http2 (show other bugs)
Version: 2.4.26
Hardware: PC Linux
: P2 regression with 6 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-06-29 11:03 UTC by Ondrej Sury
Modified: 2017-07-11 09:50 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ondrej Sury 2017-06-29 11:03:35 UTC
Hi,

several users have reported that apache2 segfaults usually in PHP when http2 module is enabled in apache2 2.4.26.

The original bugreports are here, I am copying just some traces:

~~~~
Jun 28 16:30:14 mail kernel: [707770.889003] apache2[102233]: segfault at 7f89e25c6010 ip 00007f89e1e8cde0 sp 00007fff56430e00 error 4 in libapr-1.so.0.5.2 (deleted)[7f89e1e72000+31000]
Jun 28 16:30:44 mail kernel: [707800.310376] apache2[106852]: segfault at 600000001 ip 00007fd3da2df4e2 sp 00007fd3c1d326e8 error 4 in libphp7.0.so[7fd3da0b1000+392000]
Jun 28 16:30:55 mail kernel: [707812.141281] apache2[107200]: segfault at 600000001 ip 00007fd3da2e0b04 sp 00007fd3c25359b0 error 4 in libphp7.0.so[7fd3da0b1000+392000]
Jun 28 16:32:06 mail kernel: [707882.272427] apache2[107928]: segfault at 7f6c00000048 ip 00007f6cc1ba3042 sp 00007f6ca9d5a9b0 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:32:43 mail kernel: [707919.758973] apache2[108367]: segfault at 55c900000036 ip 00007f6cc1c0057a sp 00007f6caa55b9e0 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:33:32 mail kernel: [707968.278623] apache2[108605]: segfault at 7f6c0000001a ip 00007f6cc1c00548 sp 00007f6ca95576f0 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:33:37 mail kernel: [707973.490387] apache2[108814]: segfault at 7f6e474ea5a4 ip 00007f6cc1bbdb5f sp 00007f6ca9d5a8d0 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:33:47 mail kernel: [707984.145976] apache2[107830]: segfault at 3330322e30 ip 00007f6cc1b8cb04 sp 00007f6caad5c710 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:36:04 mail kernel: [708120.747497] apache2[109092]: segfault at 600000001 ip 00007f6cc1b8cb04 sp 00007f6ca9557570 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:36:14 mail kernel: [708130.828956] apache2[109624]: segfault at 0 ip (null) sp 00007f6ca95576e8 error 14 in apache2[55c92a09e000+9e000]
Jun 28 16:37:15 mail kernel: [708191.943459] apache2[109674]: segfault at 1406 ip 00007f6cc1ba2ea1 sp 00007f6ca9557080 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:38:18 mail kernel: [708254.981575] apache2[110142]: segfault at 600000001 ip 00007f6cc1b8cb04 sp 00007f6ca9559860 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:38:21 mail kernel: [708257.266126] apache2[110454]: segfault at 60600000000 ip 00007f6cc1b8cb04 sp 00007f6ca9557290 error 4 in libphp7.0.so[7f6cc195d000+392000]
Jun 28 16:40:55 mail kernel: [708411.291971] apache2[112048]: segfault at 7fe23b4ea5ac ip 00007fe0b5b55bfa sp 00007fe09dd5a8d0 error 4 in libphp7.0.so[7fe0b58f5000+392000]
Jun 28 16:41:45 mail kernel: [708461.406343] apache2[111959]: segfault at 0 ip 00007fe0b5b3817b sp 00007fe09e55af58 error 4 in libphp7.0.so[7fe0b58f5000+392000]
Jun 28 16:44:28 mail kernel: [708624.787914] apache2[114082]: segfault at 600000001 ip 00007fe0b5b24b04 sp 00007fe09dd5a860 error 4 in libphp7.0.so[7fe0b58f5000+392000]
Jun 28 16:44:54 mail kernel: [708650.426822] apache2[114418]: segfault at 7fe23b4ea5a4 ip 00007fe0b5b55bfa sp 00007fe09e55b8d0 error 4 in libphp7.0.so[7fe0b58f5000+392000]
Jun 28 16:46:07 mail kernel: [708723.584629] apache2[114737]: segfault at 600000001 ip 00007fe0b5b24b04 sp 00007fe09d557570 error 4 in libphp7.0.so[7fe0b58f5000+392000]
Jun 28 16:47:09 mail kernel: [708785.630073] apache2[115322]: segfault at c ip 00007fe0b5b5cd67 sp 00007fe09d557110 error 4 in libphp7.0.so[7fe0b58f5000+392000]
Jun 28 16:51:56 mail kernel: [709073.089354] apache2[117357]: segfault at 7fe000000000 ip 00007fe0b5b24b04 sp 00007fe09d5597d0 error 4 in libphp7.0.so[7fe0b58f5000+392000]
~~~~

~~~~
[Thu Jun 29 06:55:23.235009 2017] [core:notice] [pid 6435] AH00051: child pid 16045 exit signal Segmentation fault (11), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': double free or corruption (fasttop): 0x00007faca40608d0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7facd0d7508b]
/lib/x86_64-linux-gnu/libc.so.6(+0x82c3a)[0x7facd0d7ec3a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7facd0d82d2c]
/usr/lib/php/20160303/xdebug.so(xdebug_llist_remove+0x4f)[0x7facc75766af]
/usr/lib/php/20160303/xdebug.so(xdebug_execute_ex+0x474)[0x7facc7566d84]
/usr/lib/apache2/modules/libphp7.1.so(zend_execute+0x188)[0x7faccd1eb458]
/usr/lib/apache2/modules/libphp7.1.so(zend_execute_scripts+0xd3)[0x7faccd1464a3]
/usr/lib/apache2/modules/libphp7.1.so(+0x30805d)[0x7faccd1ed05d]
/usr/sbin/apache2(ap_run_handler+0x40)[0x55e894d0d7c0]
/usr/sbin/apache2(ap_invoke_handler+0xc6)[0x55e894d0dd56]
/usr/sbin/apache2(ap_process_async_request+0x363)[0x55e894d25a33]
/usr/sbin/apache2(ap_process_request+0x10)[0x55e894d25b40]
/usr/lib/apache2/modules/mod_http2.so(+0x26d09)[0x7facce218d09]
/usr/sbin/apache2(ap_run_process_connection+0x40)[0x55e894d17630]
/usr/lib/apache2/modules/mod_http2.so(+0x281d7)[0x7facce21a1d7]
/usr/lib/apache2/modules/mod_http2.so(+0x2b6a3)[0x7facce21d6a3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7facd10ca6da]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7facd0e04d7f]
~~~~

~~~~
[Thu Jun 29 09:53:11.623671 2017] [core:notice] [pid 25242] AH00051: child pid 27364 exit signal Segmentation fault (11), possible coredump in /data01/apacheSegFault
[Thu Jun 29 09:53:14.973912 2017] [core:warn] [pid 25242] AH00045: child process 27665 still did not exit, sending a SIGTERM
[Thu Jun 29 09:53:16.975994 2017] [core:warn] [pid 25242] AH00045: child process 27665 still did not exit, sending a SIGTERM
[Thu Jun 29 09:53:18.978067 2017] [core:warn] [pid 25242] AH00045: child process 27665 still did not exit, sending a SIGTERM
[Thu Jun 29 09:53:20.980150 2017] [core:error] [pid 25242] AH00046: child process 27665 still did not exit, sending a SIGKILL
[Thu Jun 29 09:53:21.981314 2017] [mpm_prefork:notice] [pid 25242] AH00169: caught SIGTERM, shutting down
[Thu Jun 29 09:54:13.146363 2017] [wsgi:warn] [pid 28557] mod_wsgi: Compiled for Python/2.7.11+.
[Thu Jun 29 09:54:13.146393 2017] [wsgi:warn] [pid 28557] mod_wsgi: Runtime using Python/2.7.12.
[Thu Jun 29 09:54:13.149165 2017] [mpm_prefork:notice] [pid 28557] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2k mod_wsgi/4.5.2 Python/2.7.12 mod_perl/2.0.9 Perl/v5.22.1 configured -- resuming normal operations
~~~~
+
~~~~
[Thu Jun 29 09:47:20.978227 2017] [http2:warn] [pid 22221] [client ***.***.***.***:65284] AH10020: h2_session(29,IDLE,0): session cleanup triggered by p
ool cleanup. this should have happened earlier already.
[Thu Jun 29 09:47:20.978244 2017] [http2:warn] [pid 22221] [client ***.***.***.***:65284] AH03199: h2_session(29,IDLE,0): connection disappeared without
proper goodbye, clients will be confused, should not happen

[Thu Jun 29 09:50:41.725088 2017] [php7:error] [pid 24245] [client ***.***.***.***:55746] PHP Fatal error: Invalid opcode 137/16/8. in /data01/var/www/prod/vendor/zendframework/zendframework1/library/Zend/Config.php on line 112, referer: https://lernziele.charite.de/llpMobil/
[Thu Jun 29 09:51:20.074313 2017] [http2:warn] [pid 23933] [client ***.***.***.***:55748] AH10020: h2_session(31,IDLE,0): session cleanup triggered by pool cleanup. this should have happened earlier already.
~~~~

And here is screenshot from gdb: https://user-images.githubusercontent.com/195299/27682022-5868e1a6-5cc1-11e7-9c33-cfef81e23ad5.png


It looks like http2 module corrupts memory in other modules (or something like that).

Ondrej
Comment 1 mark burdett 2017-07-07 05:46:34 UTC
My symptoms were segfaults, run away load, and one URL actually spat out the raw PHP source code - instead of a webpage - before I shut the server down.
Comment 2 mark burdett 2017-07-07 05:48:34 UTC
Also, looking at the error log, the PHP errors indicate vital configuration required to make the app work were lost, so very unpredictable things could happen...
Comment 3 mark burdett 2017-07-09 05:28:58 UTC
.. and the solution for this was to migrate mod_php to php_fpm and prefork mpm to event mpm.
Comment 4 Stefan Eissing 2017-07-11 09:50:10 UTC
As a workaround, Apache 2.4.26 users can use the configuration directive

  H2MaxWorkers 1

to go back to the single-threaded execution model of previous httpd version.

In 2.4.27, release imminent, HTTP/2 will no longer be supported together with mpm_prefork. 

This is done in a way that is least likely interrupting existing installations. A warning will be logged and the HTTP/2 protocol will not be negotiated for any connection. Otherwise the server will work as before.

As to deep technical discussion of why the httpd team chose to disable this combination, please refer to the dev mailing list archive. Thanks.