Bug 63977 - AH00051: child pid 31483 exit signal Segmentation fault mod_proxy_html.c
Summary: AH00051: child pid 31483 exit signal Segmentation fault mod_proxy_html.c
Status: RESOLVED INVALID
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy_html (show other bugs)
Version: 2.4.26
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-29 09:11 UTC by chenkaifeng
Modified: 2019-12-02 18:59 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description chenkaifeng 2019-11-29 09:11:05 UTC

    
Comment 1 Christophe JAILLET 2019-11-29 18:41:23 UTC
Hi,

without any description at all, it is hard to diagnose anything.
Comment 2 chenkaifeng 2019-11-30 03:27:35 UTC
(In reply to Christophe JAILLET from comment #1)
> Hi,
> 
> without any description at all, it is hard to diagnose anything.

I will add description. and I am stilling checking the issue now
Comment 3 chenkaifeng 2019-11-30 04:33:16 UTC
Description:

The situation is: When we have very large html page(resource manager web,
 This page contains about 3,714 entries of application.), the httpd proxy_html  exit signal Segmentation fault.
when this page contains little entires of application, the httpd proxy_html works fine.

gdb httpd httpd core.6193
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/opt/Bigdata/Apache-httpd-2.4.26/apache2/bin/httpd -f /opt/Bigdata/Apache-httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3acec197bf in __memmove_ssse3_back () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install expat-2.1.0-10.x86_64 glibc-2.17-111.h34.x86_64 libgcc-4.8.5-4.h5.x86_64 libxml2-2.9.1-6.3.h12.x86_64 nss-softokn-freebl-3.36.0-5.h2.x86_64 xz-li-12alpha.x86_64 zlib-1.2.7-15.x86_64
(gdb) bt
#0  0x00007f3acec197bf in __memmove_ssse3_back () from /lib64/libc.so.6
#1  0x00007f3ac9a71987 in dump_content (ctx=ctx@entry=0x1360458) at mod_proxy_html.c:249
#2  0x00007f3ac9a74d35 in pendElement (ctxt=0x1360458, uname=0x133a744 "script") at mod_proxy_html.c:378
#3  0x00007f3aca42c791 in htmlParseEndTag () from /usr/lib64/libxml2.so.2
#4  0x00007f3aca4300e1 in htmlParseChunk () from /usr/lib64/libxml2.so.2
#5  0x00007f3ac9a73ab2 in proxy_html_filter (f=<optimized out>, bb=<optimized out>) at mod_proxy_html.c:915
#6  0x0000000000464cd1 in ap_process_async_request (r=0x134fe30) at http_request.c:459
#7  0x0000000000461351 in ap_process_http_async_connection (c=0x1331ec8) at http_core.c:154
#8  ap_process_http_connection (c=0x1331ec8) at http_core.c:248
#9  0x0000000000459490 in ap_run_process_connection (c=c@entry=0x1331ec8) at connection.c:42
#10 0x000000000046d10e in process_socket (my_thread_num=0, my_child_num=3, cs=0x1331e38, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:945
#11 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1849
#12 0x00007f3acf095dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3acebc094d in clone () from /lib64/libc.so.6
(gdb) info


(gdb) p ctx->buf
$25 = 0x7f3abb127010 "\n", ' ' <repeats 14 times>, "var appsTableData=[\n[\"<a href='/cluster/app/application_1574873447714_2041'>application_1574873447714_2041</a>\",\"root\",\"insert overwrite tae1128_...\\'817200\\'(Stage-1)\",\"MAPREDU"...
(gdb) p ctx->buf + len
$26 = 0x7f3abb331feb "a>\",\"0\"]\n]\n          "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to
$27 = 0x7f3abb1702a6 "873447714_2022/'>History</a>\",\"0\"],\n[\"<a href='/cluster/app/application_1574873447714_2019tion_1574873447714_2019'>application_1574873447714_2019</a>\",\"root\",\"insert e table table1128_...\\'808"...
(gdb) p len + 1 - s_from - offs
$28 = 1842525
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs
$29 = 0x7f3abb332003 <Address 0x7f3abb332003 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs^CQuit
(gdb) p len + 1 - s_from - offs
$30 = 1842525
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs -10
$31 = 0x7f3abb331ff9 "       "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 100
$32 = 0x7f3abb331f9f "ttp://node-master2gfNp:8088/proxy/application_1574873447714_0492/'>History</a>\",\"0\"]\n]\n          "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 20
$33 = 0x7f3abb331fef "\"0\"]\n]\n          "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 50
$34 = 0x7f3abb331fd1 "873447714_0492/'>History</a>\",\"0\"]\n]\n          "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 70
$35 = 0x7f3abb331fbd "oxy/application_1574873447714_0492/'>History</a>\",\"0\"]\n]\n          "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 30
$36 = 0x7f3abb331fe5 "tory</a>\",\"0\"]\n]\n          "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 10
$37 = 0x7f3abb331ff9 "       "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 5
$38 = 0x7f3abb331ffe "  "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 4
$39 = 0x7f3abb331fff " "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 3
$40 = 0x7f3abb332000 <Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 4
$41 = 0x7f3abb331fff " "<Address 0x7f3abb332000 out of bounds>
(gdb) p ctx->buf+offs+s_to + len + 1 - s_from - offs - 3
$42 = 0x7f3abb332000 <Address 0x7f3abb332000 out of bounds>
Comment 4 chenkaifeng 2019-11-30 04:34:39 UTC
The correspond code should be in function dump_content of mod_proxy_html.c file:

memmove(ctx->buf+offs+s_to, ctx->buf+offs+s_from,
                            len + 1 - s_from - offs);
Comment 5 chenkaifeng 2019-11-30 04:37:03 UTC
(gdb) info locals
m = 0x135de48
found = <optimized out>
s_from = 34
s_to = 57
match = <optimized out>
c = 0 '\000'
pmatch = {{rm_so = 540, rm_eo = 574}, {rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}, {
    rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}, {rm_so = -1, rm_eo = -1}}
subs = 0x1380998 "href='https://192.168.1.170:20026/Yarn/ResourceManager/41"
len = 2142171
offs = 299613
themap = <optimized out>
verbose = 0
Comment 6 chenkaifeng 2019-11-30 04:41:59 UTC
And below is error_log

[Fri Nov 29 17:26:02.136858 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:prog, next:ress.
[Fri Nov 29 17:26:02.136863 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136867 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136872 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:ui, next:ress.
[Fri Nov 29 17:26:02.136877 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136881 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136886 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:blac, next:klis.
[Fri Nov 29 17:26:02.136890 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136895 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136899 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136904 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:\n   , next:    .
[Fri Nov 29 17:26:02.136912 2019] [proxy_html:info] [pid 6193:tid 139890451052288] [client 192.168.1.33:49206] pcharacters, cur:text, next:/jav.
[Fri Nov 29 17:26:02.163055 2019] [proxy:debug] [pid 6193:tid 139890451052288] proxy_util.c(2171): AH00943: HTTP: has released connection for (*)
[Fri Nov 29 17:26:02.163117 2019] [proxy:debug] [pid 6193:tid 139890451052288] proxy_util.c(2994): [remote 192.168.1.210:8088] AH02642: proxy: connection shutdown
[Fri Nov 29 17:26:02.803371 2019] [core:notice] [pid 1339:tid 139890576172864] AH00051: child pid 6193 exit signal Segmentation fault (11), possible coredump in /opt/Bigdata/Apache-httpd-2.4.26/apache2
Comment 7 chenkaifeng 2019-12-02 07:25:50 UTC
have found the root cause as misconfig of proxy-html.conf
Comment 8 Christophe JAILLET 2019-12-02 18:59:20 UTC
Nice to see that you have found the root cause.

Could you share the incorrect configuration that trigger the issue?
The module should not crash only because of an incorrect config. Maybe we could issue some warning during configuration parsing, or log something at run-time.

Crashing is not really user-friendly :)