Created attachment 38316 [details] httpd 2.4.53 MPM-worker segfault backtrace Apache 2.4.53+ worker process (in MPM worker mode) crashes with a segmentation fault when processing a request with path that is equal or longer than 145 characters when virtual host has a Location directive with regex match as follows: <Location ~ "^((?!/errors/).)*$"> An example request making segfault crash (with a minimal URI length causing the crash - 145): GET /1234567890123456789001234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234 An example request working OK (with a maximal URI length NOT causing a crash - 144): GET /123456789012345678900123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123 Result of a crashed request in httpd logs: ==> /var/www/logs/error.log <== [Tue Jun 14 14:18:42.790169 2022] [core:notice] [pid 21:tid 140397196118856] AH00051: child pid 25 exit signal Segmentation fault (11), possible coredump in /tmp Crash backtrace generated from core dump attached to the issue. The request fails both in 2.4.53 and 2.4.54 e.g. Alpine-based (using PCRE 8.45): fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz (1/9) Installing libuuid (2.38-r1) (2/9) Installing apr (1.7.0-r2) (3/9) Installing expat (2.4.8-r0) (4/9) Installing apr-util (1.6.1-r12) (5/9) Installing pcre (8.45-r2) (6/9) Installing apache2 (2.4.54-r0) Executing apache2-2.4.54-r0.pre-install (7/9) Installing libpcre16 (8.45-r2) (8/9) Installing libpcre32 (8.45-r2) (9/9) Installing pcre-tools (8.45-r2) AH00292: Apache/2.4.54 (Unix) configured -- resuming normal operations but works well in old Alpine 3.13.11 bringing still 2.4.51 (using PCRE 8.43): fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz (1/9) Installing libuuid (2.34-r1) (2/9) Installing apr (1.7.0-r0) (3/9) Installing expat (2.2.9-r1) (4/9) Installing apr-util (1.6.1-r6) (5/9) Installing pcre (8.43-r1) (6/9) Installing apache2 (2.4.51-r0) Executing apache2-2.4.51-r0.pre-install (7/9) Installing libpcre16 (8.43-r1) (8/9) Installing libpcre32 (8.43-r1) (9/9) Installing pcre-tools (8.43-r1) AH00292: Apache/2.4.51 (Unix) configured -- resuming normal operations Worth noting are the following facts: 1) the issue does not occur in non-threaded MPM prefork mode, but since 2.4.53 occurs in threaded e.g. MPM worker mode 2) the pattern used in Location regexp match does not fail via pcretest call from pcre-tools (in neither version) - content of this input: /^((?!\/errors\/).)*$/I /this/is/normal/matching/path /errors/notmatching /012345678901234567890012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012 /0123456789012345678900123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123 does not fail: PCRE version 8.45 2021-06-15 /^((?!\/errors\/).)*$/I Capturing subpattern count = 1 May match empty string Options: anchored No first char No need char /this/is/normal/matching/path 0: /this/is/normal/matching/path 1: h /errors/notmatching No match /012345678901234567890012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012 0: /012345678901234567890012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012 1: 2 /0123456789012345678900123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123 0: /0123456789012345678900123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123 1: 3 3) between 2.4.52 and 2.4.53 the following change has been incorporated: ap_regex: Use Thread Local Storage (if efficient) to avoid allocations. https://github.com/apache/httpd/commit/285b6285e799d3480150375cb853d40973d3ab4c#diff-4d479e1bf17e6539c339b182d3c0861a1bd0b0ccbad5226c437d0e30f8cae0c7R395 which (among other things) also ap_regexec in server/util_pcre.c also when HAVE_PCRE2 condition is not met (e.g. ovector and nmatch/ncaps and around pcre_exec call), although please note that I am neither acquainted to this code nor am I C-lang specialist. The scripts and configuration files which might help to reproduce the issue: * run.sh - script to run httpd via Alpine in Docker: #!/bin/sh ALPINE_VERSION=${ALPINE_VERSION:-"latest"} HTTP_PORT=${HTTP_PORT:-8080} CONTAINER_NAME=${CONTAINER_NAME:-"httpd"} CONFIG_DIR=$(pwd)/httpd-config docker run \ -it \ --rm \ --name ${CONTAINER_NAME} \ --init --privileged --ulimit core=-1 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE \ -p ${HTTP_PORT}:80 \ -v ${CONFIG_DIR}/conf.d/vhost.conf:/etc/apache2/conf.d/vhost.conf:ro \ -v $(pwd)/httpd-conf-sed.txt:/mnt/httpd-conf-sed.txt:ro \ -v $(pwd)/pcretest-input.txt:/mnt/pcretest-input.txt:ro \ -v $(pwd)/tmp:/tmp \ alpine:${ALPINE_VERSION} \ sh -c 'sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t; apk add apache2 pcre-tools; pcretest /mnt/pcretest-input.txt; sed -f /mnt/httpd-conf-sed.txt -i /etc/apache2/httpd.conf; echo "CoreDumpDirectory /tmp" >> /etc/apache2/httpd.conf; httpd; tail -F /var/www/logs/*' * httpd-config/conf.d/vhost.conf - Simple vhost.conf file: ServerName segfaultserver <VirtualHost *:80> ServerName caselocation ServerAlias "*" <IfModule mod_rewrite.c> RewriteEngine on # Location pattern PCRE -> ap_location_walk -> ap_regexec_len -> pcre_exec -> libpcre.so.1 (multiple entries) -> Program terminated with signal SIGSEGV, Segmentation fault. <Location ~ "^((?!/errors/).)*$"> </Location> </IfModule> </VirtualHost> * httpd-conf-sed.txt - Sed input file to enable mpm_worker_module and rewrite_module: s#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so#\#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so# s#\#LoadModule mpm_worker_module modules/mod_mpm_worker.so#LoadModule mpm_worker_module modules/mod_mpm_worker.so# s#\#LoadModule rewrite_module modules/mod_rewrite.so#LoadModule rewrite_module modules/mod_rewrite.so# * pcretest-input.txt - PCRE test input file to verify regexp pattern works by pcretest using given version of PCRE with requests used in testing: /^((?!\/errors\/).)*$/I /this/is/normal/matching/path /errors/notmatching /012345678901234567890012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012 /0123456789012345678900123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123 * request.sh - script to make requests against started Docker container with given httpd version from Alpine: #!/bin/sh HTTP_PORT=${HTTP_PORT:-8080} CORRECT_URI="/123456789012345678900123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123" # len<=144 FAILING_URI="/1234567890123456789001234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234" # len>=145 echo "request correct URI: ${CORRECT_URI}" curl -svo /dev/null http://localhost:${HTTP_PORT}${CORRECT_URI} echo "request failing URI: ${FAILING_URI}" curl -svo /dev/null http://localhost:${HTTP_PORT}${FAILING_URI} This issue _might_ be related to another one reported for 2.4.53 and PCRE regex patterns processing - https://bz.apache.org/bugzilla/show_bug.cgi?id=66021
> <Location ~ "^((?!/errors/).)*$"> Do you create 1 capture per character on purpose, or did intend the * to come after the wildcard?
Disclaimer - I only investigated this issue, I'm not responsible for writing this regex myself. I understand that it could be rewritten to be more optimal. We have just observed that an already used httpd configuration started to fail when upgraded from 2.4.52 to 2.4.53
(In reply to Eric Covener from comment #1) > > <Location ~ "^((?!/errors/).)*$"> > > Do you create 1 capture per character on purpose, or did intend the * to > come after the wildcard? It is a similar case (regarding capture groups) as in the regexp pattern used also in https://bz.apache.org/bugzilla/show_bug.cgi?id=66021 - maybe this is why both of them are reported to fail since 2.4.53 on?
Can you please modify the regex from ^((?!/errors/).)*$ to ^(?!/errors/)(.*)$ This is not about discussing which regular expression could be better or more "correct". It should check if just the length of the string causes the failure or if it is related to the high number of captures in the first regexp with longer strings.
(In reply to Ruediger Pluem from comment #4) > Can you please modify the regex from > ^((?!/errors/).)*$ > > to > > ^(?!/errors/)(.*)$ > > This is not about discussing which regular expression could be better or > more "correct". It should check if just the length of the string causes the > failure or if it is related to the high number of captures in the first > regexp with longer strings. Thanks Ruediger, yes, thank you, the regexp in the example has been already fixed. But is has been given here as an example only to ask why it started to fail from Apache HTTPD 2.4.53 on and only in threaded MPM modes, whereas it was not failing up till 2.4.52 (in any mode of MPM).
(In reply to Krystian Nowak from comment #5) > (In reply to Ruediger Pluem from comment #4) > > Can you please modify the regex from > > ^((?!/errors/).)*$ > > > > to > > > > ^(?!/errors/)(.*)$ > > > > This is not about discussing which regular expression could be better or > > more "correct". It should check if just the length of the string causes the > > failure or if it is related to the high number of captures in the first > > regexp with longer strings. > > Thanks Ruediger, yes, thank you, the regexp in the example has been already > fixed. > But is has been given here as an example only to ask why it started to fail > from Apache HTTPD 2.4.53 on and only in threaded MPM modes, whereas it was > not failing up till 2.4.52 (in any mode of MPM). I understand, but for further investigations I would like to understand if the regexp I gave you fails in the same way. If it does not the issue might be caused by the high number of captures and thus gives a pointer where to search in the code for an issue.
(In reply to Ruediger Pluem from comment #6) > > > Can you please modify the regex from > > > ^((?!/errors/).)*$ > > > > > > to > > > > > > ^(?!/errors/)(.*)$ > > > > I understand, but for further investigations I would like to understand if > the regexp I gave you fails in the same way. If it does not the issue might > be caused by the high number of captures and thus gives a pointer where to > search in the code for an issue. I have changed the regexp according to the suggestion: +++ b/httpd-config/conf.d/vhost.conf @@ -9,7 +9,7 @@ ServerName segfaultserver RewriteEngine on # Location pattern PCRE -> ap_location_walk -> ap_regexec_len -> pcre_exec -> libpcre.so.1 (multiple entries) -> Program terminated with signal SIGSEGV, Segmentation fault. - <Location ~ "^((?!/errors/).)*$"> + <Location ~ "^(?!/errors/)(.*)$"> RewriteRule ^(.*) https://bz.apache.org%{REQUEST_URI} [R=301,L] </Location> and it works (doesn't crash any more). Hope it helps in investigation!
Created attachment 38318 [details] Honor nmatch for the vector passed to pcre_exec() Since 2.4.52 (r1898467) we always provide a cached (per-thread) buffer to pcre_exec() for the capturing groups, even though the caller does not care about them (by passing nmatch=0, which is the case of <Location> when there are no named captures). The difference is that PCRE1 treats all capturing groups as non-capturing if it's not given a vector to store the offsets, which allows for optimizations and notably less recursion (replaced by tail calls) in some cases. And the regex "^((?!/errors/).)*$" typically falls into this case where discarding/avoiding captures also avoids lot of recursion in pcre_exec() thus stack exhaustion (the crash hit here). This does not depend on the MPM, I could reproduce the crash with prefork by using a longer path. I think the correct thing to do is something like this patch which honors the "nmatch" passed by the caller as for the vector passed to PCRE1 (like before 2.4.52, with a caveat about back-references, see comment in patch), meaning no/NULL vector eventually if the caller does not care. This fixes the <Location ~ "^((?!/errors/).)*$"> case for me (and all the similar {location,directory,file,proxy,..}_walk() cases when nmatch=0 can be used) but there is nothing we can do at httpd level to prevent some (badly written) regexes from exhausting the stack in all cases, notably when captures are to be used like in the RedirectMatch case (bug 66021) or in mod_rewrite most likely. Anyway, Krystian could you test this patch for you case please?
*** Bug 66021 has been marked as a duplicate of this bug. ***
Yann, thanks for the patch! It seems it got fixed for the case described in this issue (after the patch it stopped segfaulting), but not in the case in issue 66021 (that one still fails with segfault). HTH
As I said: > but there is nothing we can do at httpd level to prevent some (badly > written) regexes from exhausting the stack in all cases, notably when > captures are to be used like in the RedirectMatch case (bug 66021) or in > mod_rewrite most likely. This is because in these cases the captures are used thus a non-NULL vector is passed to PCRE, and (a lot of) recursions happen. Doesn't it crash on 2.4.51 for e.g. RedirectMatch "^((?!/errors/).)*$" "http://www.example.com$1 ?
Got it, so at least this patch fixes the NULL vector case (as in this specific issue) - so then I can't wait being it applied and merged to 2.4 branch for next release - thanks for finding the fix for it, Yann!
And you are right Yann, in 2.4.51 it segfaults with: AH00052: child pid 44 exit signal Segmentation fault (11) when having configured: RedirectMatch "^((?!/errors/).)*$" "http://www.example.com$1 and requesting e.g.: curl -v http://localhost/subpath01/subpath02/subpath03/subpath04/subpath05/subpath06/subpath07/subpath08/subpath09/subpath10/subpath11/subpath12/subpath13/subpath14/suffix
Does it crash when 2.4.54 is compiled against PCRE2?
> Does it crash when 2.4.54 is compiled against PCRE2? No it does not, PCRE2 uses its own (re)allocation logic in the passed in pcre2_match_data context, and stack usage is better constrained. It's still possible to find a regex and subject that exhausts memory at some point I suppose though..
Additionally, Alpine's package (by default) still uses PCRE1: https://git.alpinelinux.org/aports/tree/main/apache2/APKBUILD#n15
(In reply to Yann Ylavic from comment #8) > Created attachment 38318 [details] > Honor nmatch for the vector passed to pcre_exec() Yann, will your patch be planned to get to 2.4 branch maybe?
Thanks for https://svn.apache.org/viewvc?view=revision&revision=1902732 for those pathological patterns, Yann! When +/- will you be planning to port it to 2.4 branch?