58524 – mod_http2 segfaults when under heavy load

Bug 58524 - mod_http2 segfaults when under heavy load

Summary: mod_http2 segfaults when under heavy load

Status:	RESOLVED WORKSFORME

Alias:	None

Product:	Apache httpd-2
Classification:	Unclassified
Component:	mod_http2 (show other bugs)
Version:	2.4.17
Hardware:	PC All

Importance:	P2 normal with 3 votes (vote)
Target Milestone:	---
Assignee:	Apache HTTPD Bugs Mailing List

URL:
Keywords:

Depends on:
Blocks:

Reported:	2015-10-23 08:03 UTC by Cedric Roijakkers
Modified:	2016-08-31 14:04 UTC (History)
CC List:	2 users (show)

Attachments
stability patch of mod_http2 on top of 2.4.17 (42.06 KB, patch) 2015-10-23 13:48 UTC, Stefan Eissing	Details \| Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Cedric Roijakkers 2015-10-23 08:03:16 UTC

Component versions:

httpd: 2.4.17
apr-utils: 1.5.4
apr: 1.5.2
nghttp2: 1.3.4
openssl: 1.0.2d
OS: RHEL 5.10

We've recently enabled mod_http2 in production (only protocol h2, not h2c) and this is causing segfaults. Since we use varnish and mod_proxy a lot, we were originally triggered by lots of connections staging in CLOSE_WAIT, but when looking in the logs the following segfaults could be found:

[Fri Oct 23 08:53:47.395464 2015] [core:notice] [pid 17063:tid 46986631311552] AH00052: child pid 2209 exit signal Segmentation fault (11)
[Fri Oct 23 08:54:49.418568 2015] [core:notice] [pid 17063:tid 46986631311552] AH00052: child pid 1671 exit signal Segmentation fault (11)
*** glibc detected *** /opt/appl/httpd-2.4.17/bin/httpd: double free or corruption (out): 0x00002abc4470bdd0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x32a507174f]
/lib64/libc.so.6(cfree+0x4b)[0x32a5075a4b]
/opt/appl/apr-1.5.2/lib/libapr-1.so.0(apr_allocator_destroy+0x1b)[0x2abbed566e6b]
/opt/appl/httpd/modules/mod_http2.so[0x2abbf10e082e]
/opt/appl/httpd/modules/mod_http2.so[0x2abbf10e32ee]
/opt/appl/httpd/modules/mod_http2.so[0x2abbf10db00a]
/opt/appl/httpd/modules/mod_http2.so[0x2abbf10db233]
/opt/appl/httpd/modules/mod_http2.so[0x2abbf10de351]
/opt/appl/httpd-2.4.17/bin/httpd(ap_run_process_connection+0x4a)[0x456efa]
/opt/appl/httpd-2.4.17/bin/httpd[0x468fa0]
/lib64/libpthread.so.0[0x32a5c0683d]
/lib64/libc.so.6(clone+0x6d)[0x32a50d526d]
======= Memory map: ========
00400000-00489000 r-xp 00000000 fd:0c 393227                             /opt/appl/httpd-2.4.17/bin/httpd
00688000-0068e000 rw-p 00088000 fd:0c 393227                             /opt/appl/httpd-2.4.17/bin/httpd
0068e000-00691000 rw-p 0068e000 00:00 0
14552000-16ee3000 rw-p 14552000 00:00 0                                  [heap]
32a4c00000-32a4c1c000 r-xp 00000000 fd:00 634905                         /lib64/ld-2.5.so
32a4e1c000-32a4e1d000 r--p 0001c000 fd:00 634905                         /lib64/ld-2.5.so
32a4e1d000-32a4e1e000 rw-p 0001d000 fd:00 634905                         /lib64/ld-2.5.so
32a5000000-32a514f000 r-xp 00000000 fd:00 635060                         /lib64/libc-2.5.so
32a514f000-32a534f000 ---p 0014f000 fd:00 635060                         /lib64/libc-2.5.so
32a534f000-32a5353000 r--p 0014f000 fd:00 635060                         /lib64/libc-2.5.so
32a5353000-32a5354000 rw-p 00153000 fd:00 635060                         /lib64/libc-2.5.so
32a5354000-32a5359000 rw-p 32a5354000 00:00 0
32a5400000-32a5402000 r-xp 00000000 fd:00 634901                         /lib64/libdl-2.5.so
32a5402000-32a5602000 ---p 00002000 fd:00 634901                         /lib64/libdl-2.5.so
32a5602000-32a5603000 r--p 00002000 fd:00 634901                         /lib64/libdl-2.5.so
32a5603000-32a5604000 rw-p 00003000 fd:00 634901                         /lib64/libdl-2.5.so
32a5800000-32a58f1000 r-xp 00000000 fd:00 635065                         /lib64/libdb-4.3.so
32a58f1000-32a5af1000 ---p 000f1000 fd:00 635065                         /lib64/libdb-4.3.so
32a5af1000-32a5af6000 rw-p 000f1000 fd:00 635065                         /lib64/libdb-4.3.so
32a5c00000-32a5c16000 r-xp 00000000 fd:00 635064                         /lib64/libpthread-2.5.so
32a5c16000-32a5e16000 ---p 00016000 fd:00 635064                         /lib64/libpthread-2.5.so
32a5e16000-32a5e17000 r--p 00016000 fd:00 635064                         /lib64/libpthread-2.5.so
32a5e17000-32a5e18000 rw-p 00017000 fd:00 635064                         /lib64/libpthread-2.5.so
32a5e18000-32a5e1c000 rw-p 32a5e18000 00:00 0
32a6000000-32a6007000 r-xp 00000000 fd:00 635091                         /lib64/librt-2.5.so
32a6007000-32a6207000 ---p 00007000 fd:00 635091                         /lib64/librt-2.5.so
32a6207000-32a6208000 r--p 00007000 fd:00 635091                         /lib64/librt-2.5.so
32a6208000-32a6209000 rw-p 00008000 fd:00 635091                         /lib64/librt-2.5.so
32a6400000-32a6414000 r-xp 00000000 fd:00 635249                         /lib64/libz.so.1.2.3
32a6414000-32a6613000 ---p 00014000 fd:00 635249                         /lib64/libz.so.1.2.3
32a6613000-32a6614000 rw-p 00013000 fd:00 635249                         /lib64/libz.so.1.2.3
32a7000000-32a7009000 r-xp 00000000 fd:00 635256                         /lib64/libcrypt-2.5.so
32a7009000-32a7208000 ---p 00009000 fd:00 635256                         /lib64/libcrypt-2.5.so
32a7208000-32a7209000 r--p 00008000 fd:00 635256                         /lib64/libcrypt-2.5.so
32a7209000-32a720a000 rw-p 00009000 fd:00 635256                         /lib64/libcrypt-2.5.so
32a720a000-32a7238000 rw-p 32a720a000 00:00 0
32a8800000-32a880d000 r-xp 00000000 fd:00 635253                         /lib64/libgcc_s-4.1.2-20080825.so.1
32a880d000-32a8a0d000 ---p 0000d000 fd:00 635253                         /lib64/libgcc_s-4.1.2-20080825.so.1
32a8a0d000-32a8a0e000 rw-p 0000d000 fd:00 635253                         /lib64/libgcc_s-4.1.2-20080825.so.1
32ab400000-32ab421000 r-xp 00000000 fd:00 635250                         /lib64/libexpat.so.0.5.0
32ab421000-32ab620000 ---p 00021000 fd:00 635250                         /lib64/libexpat.so.0.5.0
32ab620000-32ab623000 rw-p 00020000 fd:00 635250                         /lib64/libexpat.so.0.5.0
32abc00000-32abc11000 r-xp 00000000 fd:00 635264                         /lib64/libresolv-2.5.so
32abc11000-32abe11000 ---p 00011000 fd:00 635264                         /lib64/libresolv-2.5.so
32abe11000-32abe12000 r--p 00011000 fd:00 635264                         /lib64/libresolv-2.5.so
32abe12000-32abe13000 rw-p 00012000 fd:00 635264                         /lib64/libresolv-2.5.so
32abe13000-32abe15000 rw-p 32abe13000 00:00 0
32ac800000-32ac804000 r-xp 00000000 fd:00 635198                         /lib64/libuuid.so.1.2
32ac804000-32aca03000 ---p 00004000 fd:00 635198                         /lib64/libuuid.so.1.2
32aca03000-32aca04000 rw-p 00003000 fd:00 635198                         /lib64/libuuid.so.1.2
2abbed0c8000-2abbed0ca000 rw-p 2abbed0c8000 00:00 0
2abbed0ca000-2abbed115000 r-xp 00000000 fd:0c 82122                      /opt/appl/pcre-8.37/lib/libpcre.so.1.2.5
2abbed115000-2abbed314000 ---p 0004b000 fd:0c 82122                      /opt/appl/pcre-8.37/lib/libpcre.so.1.2.5
2abbed314000-2abbed315000 rw-p 0004a000 fd:0c 82122                      /opt/appl/pcre-8.37/lib/libpcre.so.1.2.5
2abbed315000-2abbed338000 r-xp 00000000 fd:0c 114808                     /opt/appl/apr-1.5.2/lib/libaprutil-1.so.0.5.4
2abbed338000-2abbed537000 ---p 00023000 fd:0c 114808                     /opt/appl/apr-1.5.2/lib/libaprutil-1.so.0.5.4
2abbed537000-2abbed53a000 rw-p 00022000 fd:0c 114808                     /opt/appl/apr-1.5.2/lib/libaprutil-1.so.0.5.4
2abbed54b000-2abbed54c000 rw-p 2abbed54b000 00:00 0
2abbed54c000-2abbed57b000 r-xp 00000000 fd:0c 114800                     /opt/appl/apr-1.5.2/lib/libapr-1.so.0.5.2
2abbed57b000-2abbed77a000 ---p 0002f000 fd:0c 114800                     /opt/appl/apr-1.5.2/lib/libapr-1.so.0.5.2
2abbed77a000-2abbed77c000 rw-p 0002e000 fd:0c 114800                     /opt/appl/apr-1.5.2/lib/libapr-1.so.0.5.2
2abbed77c000-2abbed780000 rw-p 2abbed77c000 00:00 0
2abbed780000-2abbed969000 r-xp 00000000 fd:0c 493184                     /opt/appl/openssl-1.0.2d/lib/libcrypto.so.1.0.0
2abbed969000-2abbedb68000 ---p 001e9000 fd:0c 493184                     /opt/appl/openssl-1.0.2d/lib/libcrypto.so.1.0.0
2abbedb68000-2abbedb90000 rw-p 001e8000 fd:0c 493184                     /opt/appl/openssl-1.0.2d/lib/libcrypto.so.1.0.0
2abbedb90000-2abbedb93000 rw-p 2abbedb90000 00:00 0
2abbedb93000-2abbedbfa000 r-xp 00000000 fd:0c 493179                     /opt/appl/openssl-1.0.2d/lib/libssl.so.1.0.0
2abbedbfa000-2abbeddf9000 ---p 00067000 fd:0c 493179                     /opt/appl/openssl-1.0.2d/lib/libssl.so.1.0.0
2abbeddf9000-2abbede04000 rw-p 00066000 fd:0c 493179                     /opt/appl/openssl-1.0.2d/lib/libssl.so.1.0.0
2abbede04000-2abbede06000 r-xp 00000000 fd:0c 394640                     /opt/appl/httpd-2.4.17/modules/mod_authz_host.so
2abbede06000-2abbee005000 ---p 00002000 fd:0c 394640                     /opt/appl/httpd-2.4.17/modules/mod_authz_host.so
2abbee005000-2abbee006000 rw-p 00001000 fd:0c 394640                     /opt/appl/httpd-2.4.17/modules/mod_authz_host.so
2abbee006000-2abbee00a000 r-xp 00000000 fd:0c 394621                     /opt/appl/httpd-2.4.17/modules/mod_authz_core.so
2abbee00a000-2abbee20a000 ---p 00004000 fd:0c 394621                     /opt/appl/httpd-2.4.17/modules/mod_authz_core.so
2abbee20a000-2abbee20b000 rw-p 00004000 fd:0c 394621                     /opt/appl/httpd-2.4.17/modules/mod_authz_core.so
2abbee20b000-2abbee20d000 r-xp 00000000 fd:0c 394647                     /opt/appl/httpd-2.4.17/modules/mod_access_compat.so
2abbee20d000-2abbee40c000 ---p 00002000 fd:0c 394647                     /opt/appl/httpd-2.4.17/modules/mod_access_compat.so
2abbee40c000-2abbee40d000 rw-p 00001000 fd:0c 394647                     /opt/appl/httpd-2.4.17/modules/mod_access_compat.so
2abbee40d000-2abbee410000 r-xp 00000000 fd:0c 394597                     /opt/appl/httpd-2.4.17/modules/mod_reqtimeout.so
2abbee410000-2abbee60f000 ---p 00003000 fd:0c 394597                     /opt/appl/httpd-2.4.17/modules/mod_reqtimeout.so
2abbee60f000-2abbee610000 rw-p 00002000 fd:0c 394597                     /opt/appl/httpd-2.4.17/modules/mod_reqtimeout.so
2abbee610000-2abbee614000 r-xp 00000000 fd:0c 394594                     /opt/appl/httpd-2.4.17/modules/mod_filter.so
2abbee614000-2abbee813000 ---p 00004000 fd:0c 394594                     /opt/appl/httpd-2.4.17/modules/mod_filter.so
2abbee813000-2abbee814000 rw-p 00003000 fd:0c 394594                     /opt/appl/httpd-2.4.17/modules/mod_filter.so
2abbee814000-2abbee818000 r-xp 00000000 fd:0c 394685                     /opt/appl/httpd-2.4.17/modules/mod_mime.so
2abbee818000-2abbeea17000 ---p 00004000 fd:0c 394685                     /opt/appl/httpd-2.4.17/modules/mod_mime.so
2abbeea17000-2abbeea18000 rw-p 00003000 fd:0c 394685                     /opt/appl/httpd-2.4.17/modules/mod_mime.so
2abbeea18000-2abbeea1e000 r-xp 00000000 fd:0c 394592                     /opt/appl/httpd-2.4.17/modules/mod_log_config.so
2abbeea1e000-2abbeec1d000 ---p 00006000 fd:0c 394592                     /opt/appl/httpd-2.4.17/modules/mod_log_config.so
2abbeec1d000-2abbeec1e000 rw-p 00005000 fd:0c 394592                     /opt/appl/httpd-2.4.17/modules/mod_log_config.so
2abbeec1e000-2abbeec1f000 r-xp 00000000 fd:0c 394657                     /opt/appl/httpd-2.4.17/modules/mod_env.so
2abbeec1f000-2abbeee1f000 ---p 00001000 fd:0c 394657                     /opt/appl/httpd-2.4.17/modules/mod_env.so
2abbeee1f000-2abbeee20000 rw-p 00001000 fd:0c 394657                     /opt/appl/httpd-2.4.17/modules/mod_env.so
2abbeee20000-2abbeee24000 r-xp 00000000 fd:0c 394617                     /opt/appl/httpd-2.4.17/modules/mod_headers.so
2abbeee24000-2abbef023000 ---p 00004000 fd:0c 394617                     /opt/appl/httpd-2.4.17/modules/mod_headers.so
2abbef023000-2abbef024000 rw-p 00003000 fd:0c 394617                     /opt/appl/httpd-2.4.17/modules/mod_headers.so
2abbef024000-2abbef027000 r-xp 00000000 fd:0c 394593                     /opt/appl/httpd-2.4.17/modules/mod_setenvif.so
2abbef027000-2abbef226000 ---p 00003000 fd:0c 394593                     /opt/appl/httpd-2.4.17/modules/mod_setenvif.so
2abbef226000-2abbef227000 rw-p 00002000 fd:0c 394593                     /opt/appl/httpd-2.4.17/modules/mod_setenvif.so
2abbef227000-2abbef229000 r-xp 00000000 fd:0c 394611                     /opt/appl/httpd-2.4.17/modules/mod_version.so
2abbef229000-2abbef428000 ---p 00002000 fd:0c 394611                     /opt/appl/httpd-2.4.17/modules/mod_version.so
2abbef428000-2abbef429000 rw-p 00001000 fd:0c 394611                     /opt/appl/httpd-2.4.17/modules/mod_version.so
2abbef429000-2abbef443000 r-xp 00000000 fd:0c 394633                     /opt/appl/httpd-2.4.17/modules/mod_proxy.so
2abbef443000-2abbef643000 ---p 0001a000 fd:0c 394633                     /opt/appl/httpd-2.4.17/modules/mod_proxy.so
2abbef643000-2abbef644000 rw-p 0001a000 fd:0c 394633                     /opt/appl/httpd-2.4.17/modules/mod_proxy.so
2abbef644000-2abbef64d000 r-xp 00000000 fd:0c 394663                     /opt/appl/httpd-2.4.17/modules/mod_proxy_http.so
2abbef64d000-2abbef84c000 ---p 00009000 fd:0c 394663                     /opt/appl/httpd-2.4.17/modules/mod_proxy_http.so
2abbef84c000-2abbef84d000 rw-p 00008000 fd:0c 394663                     /opt/appl/httpd-2.4.17/modules/mod_proxy_http.so
2abbef84d000-2abbef851000 r-xp 00000000 fd:0c 394692                     /opt/appl/httpd-2.4.17/modules/mod_proxy_connect.so
2abbef851000-2abbefa50000 ---p 00004000 fd:0c 394692                     /opt/appl/httpd-2.4.17/modules/mod_proxy_connect.so
2abbefa50000-2abbefa51000 rw-p 00003000 fd:0c 394692                     /opt/appl/httpd-2.4.17/modules/mod_proxy_connect.so
2abbefa51000-2abbefa54000 r-xp 00000000 fd:0c 394596                     /opt/appl/httpd-2.4.17/modules/mod_unixd.so
2abbefa54000-2abbefc53000 ---p 00003000 fd:0c 394596                     /opt/appl/httpd-2.4.17/modules/mod_unixd.so
2abbefc53000-2abbefc54000 rw-p 00002000 fd:0c 394596                     /opt/appl/httpd-2.4.17/modules/mod_unixd.so
2abbefc54000-2abbefc56000 r-xp 00000000 fd:0c 394603                     /opt/appl/httpd-2.4.17/modules/mod_dir.so
2abbefc56000-2abbefe55000 ---p 00002000 fd:0c 394603                     /opt/appl/httpd-2.4.17/modules/mod_dir.so
2abbefe55000-2abbefe56000 rw-p 00001000 fd:0c 394603                     /opt/appl/httpd-2.4.17/modules/mod_dir.so
2abbefe56000-2abbefe65000 r-xp 00000000 fd:0c 394609                     /opt/appl/httpd-2.4.17/modules/mod_rewrite.so
2abbefe65000-2abbf0064000 ---p 0000f000 fd:0c 394609                     /opt/appl/httpd-2.4.17/modules/mod_rewrite.so
2abbf0064000-2abbf0065000 rw-p 0000e000 fd:0c 394609                     /opt/appl/httpd-2.4.17/modules/mod_rewrite.so
2abbf0065000-2abbf0097000 r-xp 00000000 fd:0c 394606                     /opt/appl/httpd-2.4.17/modules/mod_ssl.so
2abbf0097000-2abbf0296000 ---p 00032000 fd:0c 394606                     /opt/appl/httpd-2.4.17/modules/mod_ssl.so
2abbf0296000-2abbf0299000 rw-p 00031000 fd:0c 394606                     /opt/appl/httpd-2.4.17/modules/mod_ssl.so
2abbf0299000-2abbf029b000 rw-p 2abbf0299000 00:00 0
2abbf029b000-2abbf02a0000 r-xp 00000000 fd:0c 394650                     /opt/appl/httpd-2.4.17/modules/mod_socache_shmcb.so
2abbf02a0000-2abbf049f000 ---p 00005000 fd:0c 394650                     /opt/appl/httpd-2.4.17/modules/mod_socache_shmcb.so
2abbf049f000-2abbf04a0000 rw-p 00004000 fd:0c 394650                     /opt/appl/httpd-2.4.17/modules/mod_socache_shmcb.so
2abbf04a0000-2abbf04ab000 r-xp 00000000 fd:0c 394627                     /opt/appl/httpd-2.4.17/modules/mod_include.so
2abbf04ab000-2abbf06ab000 ---p 0000b000 fd:0c 394627                     /opt/appl/httpd-2.4.17/modules/mod_include.so
2abbf06ab000-2abbf06ac000 rw-p 0000b000 fd:0c 394627                     /opt/appl/httpd-2.4.17/modules/mod_include.so
2abbf06ac000-2abbf06ae000 r-xp 00000000 fd:0c 394639                     /opt/appl/httpd-2.4.17/modules/mod_expires.so
2abbf06ae000-2abbf08ae000 ---p 00002000 fd:0c 394639                     /opt/appl/httpd-2.4.17/modules/mod_expires.so
2abbf08ae000-2abbf08af000 rw-p 00002000 fd:0c 394639                     /opt/appl/httpd-2.4.17/modules/mod_expires.so
2abbf08af000-2abbf08b6000 r-xp 00000000 fd:0c 394688                     /opt/appl/httpd-2.4.17/modules/mod_deflate.so
2abbf08b6000-2abbf0ab6000 ---p 00007000 fd:0c 394688                     /opt/appl/httpd-2.4.17/modules/mod_deflate.so
2abbf0ab6000-2abbf0ab7000 rw-p 00007000 fd:0c 394688                     /opt/appl/httpd-2.4.17/modules/mod_deflate.so
2abbf0ab7000-2abbf0abc000 r-xp 00000000 fd:0c 394646                     /opt/appl/httpd-2.4.17/modules/mod_status.so
2abbf0abc000-2abbf0cbb000 ---p 00005000 fd:0c 394646                     /opt/appl/httpd-2.4.17/modules/mod_status.so
2abbf0cbb000-2abbf0cbc000 rw-p 00004000 fd:0c 394646                     /opt/appl/httpd-2.4.17/modules/mod_status.so
2abbf0cbc000-2abbf0cbf000 r-xp 00000000 fd:0c 394673                     /opt/appl/httpd-2.4.17/modules/mod_alias.so
2abbf0cbf000-2abbf0ebe000 ---p 00003000 fd:0c 394673                     /opt/appl/httpd-2.4.17/modules/mod_alias.so
2abbf0ebe000-2abbf0ebf000 rw-p 00002000 fd:0c 394673                     /opt/appl/httpd-2.4.17/modules/mod_alias.so
2abbf0ebf000-2abbf0ed4000 r-xp 00000000 fd:0c 394591                     /opt/appl/httpd-2.4.17/modules/mod_kupale.so
2abbf0ed4000-2abbf10d4000 ---p 00015000 fd:0c 394591                     /opt/appl/httpd-2.4.17/modules/mod_kupale.so
2abbf10d4000-2abbf10d5000 rw-p 00015000 fd:0c 394591                     /opt/appl/httpd-2.4.17/modules/mod_kupale.so
2abbf10d5000-2abbf10f1000 r-xp 00000000 fd:0c 394620                     /opt/appl/httpd-2.4.17/modules/mod_http2.so
2abbf10f1000-2abbf12f1000 ---p 0001c000 fd:0c 394620                     /opt/appl/httpd-2.4.17/modules/mod_http2.so
2abbf12f1000-2abbf12f2000 rw-p 0001c000 fd:0c 394620                     /opt/appl/httpd-2.4.17/modules/mod_http2.so
2abbf12f2000-2abbf1314000 r-xp 00000000 fd:0c 98330                      /opt/appl/nghttp2-1.3.4/lib/libnghttp2.so.14.1.4
2abbf1314000-2abbf1514000 ---p 00022000 fd:0c 98330                      /opt/appl/nghttp2-1.3.4/lib/libnghttp2.so.14.1.4
2abbf1514000-2abbf1516000 rw-p 00022000 fd:0c 98330                      /opt/appl/nghttp2-1.3.4/lib/libnghttp2.so.14.1.4
2abbf1527000-2abbf1531000 r-xp 00000000 fd:00 635184                     /lib64/libnss_files-2.5.so
2abbf1531000-2abbf1730000 ---p 0000a000 fd:00 635184                     /lib64/libnss_files-2.5.so
2abbf1730000-2abbf1731000 r--p 00009000 fd:00 635184                     /lib64/libnss_files-2.5.so
2abbf1731000-2abbf1732000 rw-p 0000a000 fd:00 635184                     /lib64/libnss_files-2.5.so
2abbf1732000-2abbf17b0000 rw-s 00000000 00:09 691507410                  /dev/zero (deleted)
2abbf17b0000-2abbf182e000 rw-s 00000000 00:09 691507411                  /dev/zero (deleted)
2abbf182e000-2abbf19b3000 rw-s 00000000 00:09 682309629                  /dev/zero (deleted)
2abbf19b3000-2abbf19b4000 ---p 2abbf19b3000 00:00 0
2abbf19b4000-2abbf23b4000 rw-p 2abbf19b4000 00:00 0
2abbf23b4000-2abbf23b5000 ---p 2abbf23b4000 00:00 0
2abbf23b5000-2abbf2db5000 rw-p 2abbf23b5000 00:00 0
2abbf2db5000-2abbf2db6000 ---p 2abbf2db5000 00:00 0
2abbf2db6000-2abbf37b6000 rw-p 2abbf2db6000 00:00 0
2abbf37b6000-2abbf37b7000 ---p 2abbf37b6000 00:00 0
2abbf37b7000-2abbf41b7000 rw-p 2abbf37b7000 00:00 0
2abbf41b7000-2abbf41b8000 ---p 2abbf41b7000 00:00 0
2abbf41b8000-2abbf4bb8000 rw-p 2abbf41b8000 00:00 0
2abbf4bb8000-2abbf4bb9000 ---p 2abbf4bb8000 00:00 0
2abbf4bb9000-2abbf55b9000 rw-p 2abbf4bb9000 00:00 0
2abbf55b9000-2abbf55ba000 ---p 2abbf55b9000 00:00 0
2abbf55ba000-2abbf5fba000 rw-p 2abbf55ba000 00:00 0
2abbf5fba000-2abbf5fbb000 ---p 2abbf5fba000 00:00 0
2abbf5fbb000-2abbf69bb000 rw-p 2abbf5fbb000 00:00 0
2abbf69bb000-2abbf69bc000 ---p 2abbf69bb000 00:00 0
2abbf69bc000-2abbf73bc000 rw-p 2abbf69bc000 00:00 0
2abbf73bc000-2abbf73bd000 ---p 2abbf73bc000 00:00 0
2abbf73bd000-2abbf7dbd000 rw-p 2abbf73bd000 00:00 0
2abbf7dbd000-2abbf7dbe000 ---p 2abbf7dbd000 00:00 0
2abbf7dbe000-2abbf87be000 rw-p 2abbf7dbe000 00:00 0
2abbf87be000-2abbf87bf000 ---p 2abbf87be000 00:00 0
2abbf87bf000-2abbf91bf000 rw-p 2abbf87bf000 00:00 0
2abbf91bf000-2abbf91c0000 ---p 2abbf91bf000 00:00 0
2abbf91c0000-2abbf9bc0000 rw-p 2abbf91c0000 00:00 0
2abbf9bc0000-2abbf9bc1000 ---p 2abbf9bc0000 00:00 0
2abbf9bc1000-2abbfa5c1000 rw-p 2abbf9bc1000 00:00 0
2abbfa5c1000-2abbfa5c2000 ---p 2abbfa5c1000 00:00 0
2abbfa5c2000-2abbfafc2000 rw-p 2abbfa5c2000 00:00 0
2abbfafc2000-2abbfafc3000 ---p 2abbfafc2000 00:00 0
2abbfafc3000-2abbfb9c3000 rw-p 2abbfafc3000 00:00 0
2abbfb9c3000-2abbfb9c4000 ---p 2abbfb9c3000 00:00 0
2abbfb9c4000-2abbfc3c4000 rw-p 2abbfb9c4000 00:00 0
2abbfc3c4000-2abbfc3c5000 ---p 2abbfc3c4000 00:00 0
2abbfc3c5000-2abbfcdc5000 rw-p 2abbfc3c5000 00:00 0
2abbfcdc5000-2abbfcdc6000 ---p 2abbfcdc5000 00:00 0
2abbfcdc6000-2abbfd7c6000 rw-p 2abbfcdc6000 00:00 0
2abbfd7c6000-2abbfd7c7000 ---p 2abbfd7c6000 00:00 0
2abbfd7c7000-2abbfe1c7000 rw-p 2abbfd7c7000 00:00 0
2abbfe1c7000-2abbfe1c8000 ---p 2abbfe1c7000 00:00 0
2abbfe1c8000-2abbfebc8000 rw-p 2abbfe1c8000 00:00 0
2abbfebc8000-2abbfebc9000 ---p 2abbfebc8000 00:00 0
2abbfebc9000-2abbff5c9000 rw-p 2abbfebc9000 00:00 0
2abbff5c9000-2abbff5ca000 ---p 2abbff5c9000 00:00 0
2abbff5ca000-2abbfffca000 rw-p 2abbff5ca000 00:00 0
2abbfffca000-2abbfffcb000 ---p 2abbfffca000 00:00 0
2abbfffcb000-2abc009cb000 rw-p 2abbfffcb000 00:00 0
2abc009cb000-2abc009cc000 ---p 2abc009cb000 00:00 0
2abc009cc000-2abc013cc000 rw-p 2abc009cc000 00:00 0
2abc013cc000-2abc013cd000 ---p 2abc013cc000 00:00 0
2abc013cd000-2abc01dcd000 rw-p 2abc013cd000 00:00 0
2abc01dcd000-2abc01dce000 ---p 2abc01dcd000 00:00 0
2abc01dce000-2abc027ce000 rw-p 2abc01dce000 00:00 0
2abc027ce000-2abc027cf000 ---p 2abc027ce000 00:00 0
2abc027cf000-2abc031cf000 rw-p 2abc027cf000 00:00 0
2abc031cf000-2abc031d0000 ---p 2abc031cf000 00:00 0
2abc031d0000-2abc03bd0000 rw-p 2abc031d0000 00:00 0
2abc03bd0000-2abc03bd1000 ---p 2abc03bd0000 00:00 0
2abc03bd1000-2abc045d1000 rw-p 2abc03bd1000 00:00 0
2abc045d1000-2abc045d2000 ---p 2abc045d1000 00:00 0
2abc045d2000-2abc04fd2000 rw-p 2abc045d2000 00:00 0
2abc04fd2000-2abc04fd3000 ---p 2abc04fd2000 00:00 0
2abc04fd3000-2abc059d3000 rw-p 2abc04fd3000 00:00 0
2abc059d3000-2abc059d4000 ---p 2abc059d3000 00:00 0
2abc059d4000-2abc063d4000 rw-p 2abc059d4000 00:00 0
2abc063d4000-2abc063d5000 ---p 2abc063d4000 00:00 0
2abc063d5000-2abc06dd5000 rw-p 2abc063d5000 00:00 0
2abc06dd5000-2abc06dd6000 ---p 2abc06dd5000 00:00 0
2abc06dd6000-2abc077d6000 rw-p 2abc06dd6000 00:00 0
2abc077d6000-2abc077d7000 ---p 2abc077d6000 00:00 0
2abc077d7000-2abc081d7000 rw-p 2abc077d7000 00:00 0
2abc081d7000-2abc081d8000 ---p 2abc081d7000 00:00 0
2abc081d8000-2abc08bd8000 rw-p 2abc081d8000 00:00 0
2abc08bd8000-2abc08bd9000 ---p 2abc08bd8000 00:00 0
2abc08bd9000-2abc095d9000 rw-p 2abc08bd9000 00:00 0
2abc095d9000-2abc095da000 ---p 2abc095d9000 00:00 0
2abc095da000-2abc09fda000 rw-p 2abc095da000 00:00 0
2abc09fda000-2abc09fdb000 ---p 2abc09fda000 00:00 0
2abc09fdb000-2abc0a9db000 rw-p 2abc09fdb000 00:00 0
2abc0a9db000-2abc0a9dc000 ---p 2abc0a9db000 00:00 0
2abc0a9dc000-2abc0b3dc000 rw-p 2abc0a9dc000 00:00 0
2abc0b3dc000-2abc0b3dd000 ---p 2abc0b3dc000 00:00 0
2abc0b3dd000-2abc0bddd000 rw-p 2abc0b3dd000 00:00 0
2abc0bddd000-2abc0bdde000 ---p 2abc0bddd000 00:00 0
2abc0bdde000-2abc0c7de000 rw-p 2abc0bdde000 00:00 0
2abc0c7de000-2abc0c7df000 ---p 2abc0c7de000 00:00 0
2abc0c7df000-2abc0d1df000 rw-p 2abc0c7df000 00:00 0
2abc0d1df000-2abc0d1e0000 ---p 2abc0d1df000 00:00 0
2abc0d1e0000-2abc0dbe0000 rw-p 2abc0d1e0000 00:00 0
2abc0dbe0000-2abc0dbe1000 ---p 2abc0dbe0000 00:00 0
2abc0dbe1000-2abc0e5e1000 rw-p 2abc0dbe1000 00:00 0
2abc0e5e1000-2abc0e5e2000 ---p 2abc0e5e1000 00:00 0
2abc0e5e2000-2abc0efe2000 rw-p 2abc0e5e2000 00:00 0
2abc0efe2000-2abc0efe3000 ---p 2abc0efe2000 00:00 0
2abc0efe3000-2abc0f9e3000 rw-p 2abc0efe3000 00:00 0
2abc0f9e3000-2abc0f9e4000 ---p 2abc0f9e3000 00:00 0
2abc0f9e4000-2abc103e4000 rw-p 2abc0f9e4000 00:00 0
2abc103e4000-2abc103e5000 ---p 2abc103e4000 00:00 0
2abc103e5000-2abc10de5000 rw-p 2abc103e5000 00:00 0
2abc10de5000-2abc10de6000 ---p 2abc10de5000 00:00 0
2abc10de6000-2abc117e6000 rw-p 2abc10de6000 00:00 0
2abc117e6000-2abc117e7000 ---p 2abc117e6000 00:00 0
2abc117e7000-2abc121e7000 rw-p 2abc117e7000 00:00 0
2abc121e7000-2abc121e8000 ---p 2abc121e7000 00:00 0
2abc121e8000-2abc12be8000 rw-p 2abc121e8000 00:00 0
2abc12be8000-2abc12be9000 ---p 2abc12be8000 00:00 0
2abc12be9000-2abc135e9000 rw-p 2abc12be9000 00:00 0
2abc135e9000-2abc135ea000 ---p 2abc135e9000 00:00 0
2abc135ea000-2abc13fea000 rw-p 2abc135ea000 00:00 0
2abc13fea000-2abc13feb000 ---p 2abc13fea000 00:00 0
2abc13feb000-2abc149eb000 rw-p 2abc13feb000 00:00 0
2abc149eb000-2abc149ec000 ---p 2abc149eb000 00:00 0
2abc149ec000-2abc153ec000 rw-p 2abc149ec000 00:00 0
2abc153ec000-2abc153ed000 ---p 2abc153ec000 00:00 0
2abc153ed000-2abc15ded000 rw-p 2abc153ed000 00:00 0
2abc15ded000-2abc15dee000 ---p 2abc15ded000 00:00 0
2abc15dee000-2abc167ee000 rw-p 2abc15dee000 00:00 0
2abc167ee000-2abc167ef000 ---p 2abc167ee000 00:00 0
2abc167ef000-2abc171ef000 rw-p 2abc167ef000 00:00 0
2abc171ef000-2abc171f0000 ---p 2abc171ef000 00:00 0
2abc171f0000-2abc17bf0000 rw-p 2abc171f0000 00:00 0
2abc17bf0000-2abc17bf1000 ---p 2abc17bf0000 00:00 0
2abc17bf1000-2abc185f1000 rw-p 2abc17bf1000 00:00 0
2abc185f1000-2abc185f2000 ---p 2abc185f1000 00:00 0
2abc185f2000-2abc18ff2000 rw-p 2abc185f2000 00:00 0
2abc18ff2000-2abc18ff3000 ---p 2abc18ff2000 00:00 0
2abc18ff3000-2abc199f3000 rw-p 2abc18ff3000 00:00 0
2abc1dffa000-2abc1dffb000 ---p 2abc1dffa000 00:00 0
2abc1dffb000-2abc1e9fb000 rw-p 2abc1dffb000 00:00 0
2abc1e9fb000-2abc1e9fc000 ---p 2abc1e9fb000 00:00 0
2abc1e9fc000-2abc1f3fc000 rw-p 2abc1e9fc000 00:00 0
2abc1f3fc000-2abc1f3fd000 ---p 2abc1f3fc000 00:00 0
2abc1f3fd000-2abc1fdfd000 rw-p 2abc1f3fd000 00:00 0
2abc1fdfd000-2abc1fdfe000 ---p 2abc1fdfd000 00:00 0
2abc1fdfe000-2abc207fe000 rw-p 2abc1fdfe000 00:00 0
2abc207fe000-2abc207ff000 ---p 2abc207fe000 00:00 0
2abc207ff000-2abc211ff000 rw-p 2abc207ff000 00:00 0
2abc211ff000-2abc21200000 ---p 2abc211ff000 00:00 0
2abc21200000-2abc21c00000 rw-p 2abc21200000 00:00 0
2abc21c00000-2abc21c01000 ---p 2abc21c00000 00:00 0
2abc21c01000-2abc22601000 rw-p 2abc21c01000 00:00 0
2abc22601000-2abc22602000 ---p 2abc22601000 00:00 0
2abc22602000-2abc23002000 rw-p 2abc22602000 00:00 0
2abc23002000-2abc23003000 ---p 2abc23002000 00:00 0
2abc23003000-2abc23a03000 rw-p 2abc23003000 00:00 0
2abc23a03000-2abc23a04000 ---p 2abc23a03000 00:00 0
2abc23a04000-2abc24404000 rw-p 2abc23a04000 00:00 0
2abc24404000-2abc24405000 ---p 2abc24404000 00:00 0
2abc24405000-2abc24e05000 rw-p 2abc24405000 00:00 0
2abc24e05000-2abc24e06000 ---p 2abc24e05000 00:00 0
2abc24e06000-2abc25806000 rw-p 2abc24e06000 00:00 0
2abc25806000-2abc25807000 ---p 2abc25806000 00:00 0
2abc25807000-2abc26207000 rw-p 2abc25807000 00:00 0
2abc26207000-2abc26208000 ---p 2abc26207000 00:00 0
2abc26208000-2abc26c08000 rw-p 2abc26208000 00:00 0
2abc26c08000-2abc26c09000 ---p 2abc26c08000 00:00 0
2abc26c09000-2abc27609000 rw-p 2abc26c09000 00:00 0
2abc27609000-2abc2760a000 ---p 2abc27609000 00:00 0
2abc2760a000-2abc2800a000 rw-p 2abc2760a000 00:00 0
2abc2800a000-2abc2800b000 ---p 2abc2800a000 00:00 0
2abc2800b000-2abc28a0b000 rw-p 2abc2800b000 00:00 0
2abc28a0b000-2abc28a0c000 ---p 2abc28a0b000 00:00 0
2abc28a0c000-2abc2940c000 rw-p 2abc28a0c000 00:00 0
2abc2940c000-2abc2940d000 ---p 2abc2940c000 00:00 0
2abc2940d000-2abc29e0d000 rw-p 2abc2940d000 00:00 0
2abc29e0d000-2abc29e0e000 ---p 2abc29e0d000 00:00 0
2abc29e0e000-2abc2a80e000 rw-p 2abc29e0e000 00:00 0
2abc2a80e000-2abc2a80f000 ---p 2abc2a80e000 00:00 0
2abc2a80f000-2abc2b20f000 rw-p 2abc2a80f000 00:00 0
2abc2b20f000-2abc2b210000 ---p 2abc2b20f000 00:00 0
2abc2b210000-2abc2bc10000 rw-p 2abc2b210000 00:00 0
2abc2bc10000-2abc2bc11000 ---p 2abc2bc10000 00:00 0
2abc2bc11000-2abc2c611000 rw-p 2abc2bc11000 00:00 0
2abc2c611000-2abc2c612000 ---p 2abc2c611000 00:00 0
2abc2c612000-2abc2d012000 rw-p 2abc2c612000 00:00 0
2abc2d012000-2abc2d013000 ---p 2abc2d012000 00:00 0
2abc2d013000-2abc2da13000 rw-p 2abc2d013000 00:00 0
2abc2da13000-2abc2da14000 ---p 2abc2da13000 00:00 0
2abc2da14000-2abc2e414000 rw-p 2abc2da14000 00:00 0
2abc2e414000-2abc2e415000 ---p 2abc2e414000 00:00 0
2abc2e415000-2abc2ee15000 rw-p 2abc2e415000 00:00 0
2abc2ee15000-2abc2ee16000 ---p 2abc2ee15000 00:00 0
2abc2ee16000-2abc2f816000 rw-p 2abc2ee16000 00:00 0
2abc2f816000-2abc2f817000 ---p 2abc2f816000 00:00 0
2abc2f817000-2abc30217000 rw-p 2abc2f817000 00:00 0
2abc30217000-2abc30218000 ---p 2abc30217000 00:00 0
2abc30218000-2abc30c18000 rw-p 2abc30218000 00:00 0
2abc30c18000-2abc30c19000 ---p 2abc30c18000 00:00 0
2abc30c19000-2abc31619000 rw-p 2abc30c19000 00:00 0
2abc31619000-2abc3161a000 ---p 2abc31619000 00:00 0
2abc3161a000-2abc3201a000 rw-p 2abc3161a000 00:00 0
2abc3201a000-2abc3201b000 ---p 2abc3201a000 00:00 0
2abc3201b000-2abc32a1b000 rw-p 2abc3201b000 00:00 0
2abc32a1b000-2abc32a1c000 ---p 2abc32a1b000 00:00 0
2abc32a1c000-2abc3341c000 rw-p 2abc32a1c000 00:00 0
2abc3341c000-2abc3341d000 ---p 2abc3341c000 00:00 0
2abc3341d000-2abc33e1d000 rw-p 2abc3341d000 00:00 0
2abc33e1d000-2abc33e1e000 ---p 2abc33e1d000 00:00 0
2abc33e1e000-2abc3481e000 rw-p 2abc33e1e000 00:00 0
2abc3481e000-2abc3481f000 ---p 2abc3481e000 00:00 0
2abc3481f000-2abc3521f000 rw-p 2abc3481f000 00:00 0
2abc3521f000-2abc35220000 ---p 2abc3521f000 00:00 0
2abc35220000-2abc35c20000 rw-p 2abc35220000 00:00 0
2abc35c20000-2abc35c21000 ---p 2abc35c20000 00:00 0
2abc35c21000-2abc36621000 rw-p 2abc35c21000 00:00 0
2abc36621000-2abc36622000 ---p 2abc36621000 00:00 0
2abc36622000-2abc37022000 rw-p 2abc36622000 00:00 0
2abc37022000-2abc37023000 ---p 2abc37022000 00:00 0
2abc37023000-2abc37a23000 rw-p 2abc37023000 00:00 0
2abc37a23000-2abc37a24000 ---p 2abc37a23000 00:00 0
2abc37a24000-2abc38424000 rw-p 2abc37a24000 00:00 0
2abc38424000-2abc38425000 ---p 2abc38424000 00:00 0
2abc38425000-2abc38e25000 rw-p 2abc38425000 00:00 0
2abc38e25000-2abc38e26000 ---p 2abc38e25000 00:00 0
2abc38e26000-2abc39826000 rw-p 2abc38e26000 00:00 0
2abc39826000-2abc39827000 ---p 2abc39826000 00:00 0
2abc39827000-2abc3a227000 rw-p 2abc39827000 00:00 0
2abc3a227000-2abc3a228000 ---p 2abc3a227000 00:00 0
2abc3a228000-2abc3ac28000 rw-p 2abc3a228000 00:00 0
2abc3ac28000-2abc3ac29000 ---p 2abc3ac28000 00:00 0
2abc3ac29000-2abc3b629000 rw-p 2abc3ac29000 00:00 0
2abc3b629000-2abc3b62a000 ---p 2abc3b629000 00:00 0
2abc3b62a000-2abc3c02a000 rw-p 2abc3b62a000 00:00 0
2abc3c02a000-2abc3c02b000 ---p 2abc3c02a000 00:00 0
2abc3c02b000-2abc3ca2b000 rw-p 2abc3c02b000 00:00 0
2abc3ca2b000-2abc3ca2c000 ---p 2abc3ca2b000 00:00 0
2abc3ca2c000-2abc3d42c000 rw-p 2abc3ca2c000 00:00 0
2abc3d42c000-2abc3d42d000 ---p 2abc3d42c000 00:00 0
2abc3d42d000-2abc3de2d000 rw-p 2abc3d42d000 00:00 0
2abc3de2d000-2abc3de2e000 ---p 2abc3de2d000 00:00 0
2abc3de2e000-2abc3e82e000 rw-p 2abc3de2e000 00:00 0
2abc3e82e000-2abc3e82f000 ---p 2abc3e82e000 00:00 0
2abc3e82f000-2abc3f22f000 rw-p 2abc3e82f000 00:00 0
2abc3f22f000-2abc3f230000 ---p 2abc3f22f000 00:00 0
2abc3f230000-2abc3fc30000 rw-p 2abc3f230000 00:00 0
2abc3fc30000-2abc3fc31000 ---p 2abc3fc30000 00:00 0
2abc3fc31000-2abc40631000 rw-p 2abc3fc31000 00:00 0
2abc40631000-2abc40632000 ---p 2abc40631000 00:00 0
2abc40632000-2abc41032000 rw-p 2abc40632000 00:00 0
2abc41032000-2abc41033000 ---p 2abc41032000 00:00 0
2abc41033000-2abc41a33000 rw-p 2abc41033000 00:00 0
2abc41a33000-2abc41a34000 ---p 2abc41a33000 00:00 0
2abc41a34000-2abc42434000 rw-p 2abc41a34000 00:00 0
2abc42e46000-2abc42e4a000 r-xp 00000000 fd:00 634903                     /lib64/libnss_dns-2.5.so
2abc42e4a000-2abc43049000 ---p 00004000 fd:00 634903                     /lib64/libnss_dns-2.5.so
2abc43049000-2abc4304a000 r--p 00003000 fd:00 634903                     /lib64/libnss_dns-2.5.so
2abc4304a000-2abc4304b000 rw-p 00004000 fd:00 634903                     /lib64/libnss_dns-2.5.so
2abc44000000-2abc44953000 rw-p 2abc44000000 00:00 0
2abc44953000-2abc48000000 ---p 2abc44953000 00:00 0
7fffa8740000-7fffa8756000 rw-p 7ffffffe8000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]
[Fri Oct 23 09:05:24.611786 2015] [core:notice] [pid 17063:tid 46986631311552] AH00052: child pid 3282 exit signal Aborted (6)
[Fri Oct 23 09:12:02.600072 2015] [http2:error] [pid 7881:tid 46987844122944] (70007)The timeout specified has expired: [client 82.73.230.49:14157] AH02925: h2_stream(747-65): failed send_data_cb
[Fri Oct 23 09:12:59.741394 2015] [core:notice] [pid 17063:tid 46986631311552] AH00052: child pid 8672 exit signal Segmentation fault (11)

Probably this connection failed at the moment, but there were no real production issues, except for the connection count going up. Disabling mod_http2 and restarting the server solved the issue, but of course also leaves us with no http/2 support.

Other production sites with less load seemed not to have this issue. Our main production site (average 7,5 million hits per day per server) has this issue, while other servers (500k hits per day per server) do not have the issue, running identical software builds on the same OS).

Comment 1 Stefan Eissing 2015-10-23 12:01:01 UTC

Thanks for the detailed information. I analyzed this together with Yann Ylavic and we found two places in mod_http2 where the module could veer into non-deterministic behavior under high load.

If you are interested, I can create a patch for 2.4.x with the change currently applied to trunk.

//Stefan

Comment 2 Cedric Roijakkers 2015-10-23 12:15:21 UTC

Please do create the patch, I will give it a try :-)

Comment 3 Stefan Eissing 2015-10-23 13:48:32 UTC

Created attachment 33199 [details]
stability patch of mod_http2 on top of 2.4.17

Stability improvements of mod_http2 in trunk, without changes to other httpd parts for httpd 2.4.17.

Comment 4 Stefan Eissing 2015-10-23 13:48:49 UTC

Here, the patch. May it server you well!

Comment 5 Cedric Roijakkers 2015-10-27 13:24:17 UTC

Hi, I have applied your patch on one production server and left it running there for some time.

I'm now getting less error logging, but still the occasional segfault:

[Tue Oct 27 14:13:08.126175 2015] [core:notice] [pid 26297:tid 47021196566720] AH00052: child pid 26583 exit signal Segmentation fault (11)
[Tue Oct 27 14:13:54.135349 2015] [core:notice] [pid 26297:tid 47021196566720] AH00052: child pid 26305 exit signal Segmentation fault (11)

Is there any logging that I can switch on that would help you?

Interestingly enough, I still see lots of connections stuck in CLOSE_WAIT while the mod_http2 is running. On the other production servers, at the same time, there are around 10-15 connections in CLOSE_WAIT, with mod_http2 enabled the value hangs between 200 and 300, which still indicates something wrong.

The reason why we have so many would probably be because our setup includes a lot of mod_proxy usage, we proxy to varnish, which proxies back to apache after applying caching. For a subset of calls, we then later proxy to Tomcat, so there is quite a lot of proxying going on, and when a process dies, I can imagine the connection stays open in CLOSE_WAIT. Restarting apache immediately closes all the CLOSE_WAIT connections.

Comment 6 Stefan Eissing 2015-10-27 14:33:50 UTC

Thanks for the feedback. Are you able to provide me a backtrace as you did before on the patched version? Curious if we are still crashing in apr_allocator_destroy.

Also, which mpm do you run on?

Re: CLOSE WAIT. I have in the meantime made some improvements for stream cleanup which you may find in https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha
in case you want to help testing. Please understand that this is alpha code I use together with other sites to track down issues and verify new features.

//Stefan

Comment 7 Cedric Roijakkers 2015-10-27 14:47:04 UTC

Is there a way to enable the backtrace in the logging? Last time it appeared all by itself and this time it didn't in the error.log. Maybe I'm missing something?

I will try your patch, but yes, I do understand this is quite risky...

Comment 8 Stefan Eissing 2016-08-29 07:58:31 UTC

Is this report still of interest in the current released versions? Otherwise I will close this issue. Thanks.

Comment 9 Stefan Eissing 2016-08-31 14:04:27 UTC

Not reproducible in current release.