Bug 64533 - Http crashes observed during fuzzing testing
Summary: Http crashes observed during fuzzing testing
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.46
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-18 05:48 UTC by wei-mark.zheng@nokia-sbell.com
Modified: 2021-04-18 17:08 UTC (History)
1 user (show)



Attachments
coredump (752.02 KB, application/x-zip-compressed)
2020-06-18 05:55 UTC, wei-mark.zheng@nokia-sbell.com
Details
coredump_2 (538.17 KB, application/x-zip-compressed)
2020-06-18 05:55 UTC, wei-mark.zheng@nokia-sbell.com
Details
backtrace (15.53 KB, text/plain)
2020-06-23 01:21 UTC, wei-mark.zheng@nokia-sbell.com
Details
configuration files (18.40 KB, application/x-zip-compressed)
2020-06-23 01:21 UTC, wei-mark.zheng@nokia-sbell.com
Details
debug logs with 2.4.43 (4.16 KB, application/x-zip-compressed)
2020-07-09 07:49 UTC, wei-mark.zheng@nokia-sbell.com
Details
backtraces_0710 (54.78 KB, text/plain)
2020-07-10 07:38 UTC, wei-mark.zheng@nokia-sbell.com
Details
backstraces_2909 (186.15 KB, application/x-zip-compressed)
2020-09-29 09:28 UTC, wei-mark.zheng@nokia-sbell.com
Details
backtraces_0930 (702.18 KB, application/x-zip-compressed)
2020-09-30 02:21 UTC, wei-mark.zheng@nokia-sbell.com
Details
backtraces_0910 (17.62 KB, application/x-zip-compressed)
2020-10-09 09:10 UTC, wei-mark.zheng@nokia-sbell.com
Details

Note You need to log in before you can comment on or make changes to this bug.
Description wei-mark.zheng@nokia-sbell.com 2020-06-18 05:48:42 UTC
http crashes was observed during the fuzzing testing. The http version is Apache/2.4.41 (Unix). See logs attached. Please check it what is the cause of this problem.

   Note: Fuzzing testing is to send malformed packets targets to the service to verify the http service robustness under this situation.
Comment 1 wei-mark.zheng@nokia-sbell.com 2020-06-18 05:55:13 UTC
Created attachment 37314 [details]
coredump
Comment 2 wei-mark.zheng@nokia-sbell.com 2020-06-18 05:55:49 UTC
Created attachment 37315 [details]
coredump_2
Comment 3 Ruediger Pluem 2020-06-18 07:32:30 UTC
We need:

1. Stacktraces from gdb (see http://httpd.apache.org/dev/debugging.html#crashes). The coredumps need to be analyzed on the system where they got created.
2. We need the error and access logs that were recorded during the crash. The more verbose the error logs the better.
3. We need the configuration used during the test.
Comment 4 wei-mark.zheng@nokia-sbell.com 2020-06-23 01:21:03 UTC
Created attachment 37323 [details]
backtrace
Comment 5 wei-mark.zheng@nokia-sbell.com 2020-06-23 01:21:24 UTC
Created attachment 37324 [details]
configuration files
Comment 6 wei-mark.zheng@nokia-sbell.com 2020-06-23 01:21:32 UTC
The backtrace and configuration files attached for your further checking. Please let us know if anything needed.
Comment 7 Ruediger Pluem 2020-06-23 06:20:27 UTC
Unfortunately the stacktraces do not help as they are not complete. Please try to install debugging symbols for APR / APR-UTIL and httpd as well.
Comment 8 wei-mark.zheng@nokia-sbell.com 2020-07-09 07:49:31 UTC
Created attachment 37355 [details]
debug logs with 2.4.43

we have reproduced the issue and collected the logs again with http 2.4.43.
Attached the tar file which has details. Please check.
Comment 9 Ruediger Pluem 2020-07-09 18:55:32 UTC
Unfortunately the stacktraces are still incomplete.
Comment 10 wei-mark.zheng@nokia-sbell.com 2020-07-10 04:41:59 UTC
Thanks for the feedback.
We are not sure what step is missing during the stacktraces capture.
Could you please kindly give some guidelines on the procedure to capture a full stacktraces for your analysis ? Thanks.
Comment 11 wei-mark.zheng@nokia-sbell.com 2020-07-10 04:46:58 UTC
BTW: could you share your mailaddress, then it would be more effective to discuss directly in the mail.
Comment 12 wei-mark.zheng@nokia-sbell.com 2020-07-10 07:38:58 UTC
Created attachment 37358 [details]
backtraces_0710
Comment 13 wei-mark.zheng@nokia-sbell.com 2020-07-10 07:42:14 UTC
Latest backtraces attached. This is following the guidelines http://httpd.apache.org/dev/debugging.html#crashes. If this is  no completed traces, then can we have a virtual-meeting or mail discussion to discuss how to proceeded for us ? Thanks.
Comment 14 wei-mark.zheng@nokia-sbell.com 2020-08-21 06:28:19 UTC
any updates on this case ?
Comment 15 Ruediger Pluem 2020-08-21 07:32:25 UTC
Unfortunateley the backtraces are inclomplete and do not tell us where the crash happens. I don't know why the instructions on http://httpd.apache.org/dev/debugging.html#crashes do not deliver correct backtraces. It is likely related to your Linux setup. Which Linux distro are you using?
Comment 16 wei-mark.zheng@nokia-sbell.com 2020-08-27 06:21:48 UTC
Hi Ruediger Pluem,
   The linux distro we are using is linux MIPS64, kernel version 4.4.227.

[root@CFPU-0(RNC-1009) /root]
 # uname -ar
Linux CFPU-0 4.4.227-octeon-distro.git-v2.105-2-rc-wnd #1 SMP Fri Jul 31 11:39:15 UTC 2020 mips64 mips64 mips64 GNU/Linux
Comment 17 Ruediger Pluem 2020-08-27 07:00:04 UTC
Are you able to reproduce the same issue with a different Linux distribution?
Comment 18 wei-mark.zheng@nokia-sbell.com 2020-08-27 07:05:37 UTC
Yes, we have observed the crash starting from the Apache/2.4.41 which is based on another old kernel version.  
But this issue was not observed from old Apache version until Apache/2.4.39.
Comment 19 wei-mark.zheng@nokia-sbell.com 2020-09-01 07:22:25 UTC
Hi Ruediger Pluem,
  Can you share any debug options which can be used for compiling httpd which can further be used to get more information ? 
  Also could it possible for you to have remote debugging session where you can login to our environment directly ? Thanks.
Comment 20 Ruediger Pluem 2020-09-01 10:07:48 UTC
As said, you should try to reproduce the issue on a more common Linux distribution (Debian or RedHat based) and probably on non MIPS hardware. It looks like your current setup is not capable of producing the debug information needed for further investigations.
Comment 21 wei-mark.zheng@nokia-sbell.com 2020-09-08 09:07:38 UTC
Hi Ruediger Pluem,
   I understood your suggestion. Here the situation is that our product is based on MIPS hardware and linux distribution for quite long time. And we are using apache component from the start of our product when it works fine without issue.
The problem occurs starting from version 2.4.41 and remains still now. So it is likely the problem was introduced in version 2.4.41 by new software code.
  Some questions:
1. Do you receive similar issue from other products using same apache version ?
2. Is there any other possible way to help debug this issue ?
  Please kindly suggest, thanks !
Comment 22 Ruediger Pluem 2020-09-08 11:04:05 UTC
(In reply to wei-mark.zheng@nokia-sbell.com from comment #21)
> Hi Ruediger Pluem,
>    I understood your suggestion. Here the situation is that our product is
> based on MIPS hardware and linux distribution for quite long time. And we
> are using apache component from the start of our product when it works fine
> without issue.
> The problem occurs starting from version 2.4.41 and remains still now. So it
> is likely the problem was introduced in version 2.4.41 by new software code.
>   Some questions:
> 1. Do you receive similar issue from other products using same apache
> version ?

No.

> 2. Is there any other possible way to help debug this issue ?
>   Please kindly suggest, thanks !

I am out of ideas for your platform. Hence the idea of you doing your fuzzing against a more common platform where we could possibly get the data we need if the issue is reproducable there.
Comment 23 wei-mark.zheng@nokia-sbell.com 2020-09-23 10:06:02 UTC
We have observed the crash for new 2.4.46 version as well.
Server version: Apache/2.4.46 (Fedora) Server

In addition, the crash is seen when https [Port 80] and TLS [port 443] codenomicon suites are run together which sending malformed packets continously.
But no crash observed when both the suites are run separately.
Comment 24 Ruediger Pluem 2020-09-23 14:32:43 UTC
(In reply to wei-mark.zheng@nokia-sbell.com from comment #23)
> We have observed the crash for new 2.4.46 version as well.
> Server version: Apache/2.4.46 (Fedora) Server

Looks like you are now running your fuzzing tests against a Fedora build and not your Linux MIPS distro. If this is the case can you please provide stacktraces again. Hopefully they are usable then.
Comment 25 wei-mark.zheng@nokia-sbell.com 2020-09-29 09:28:35 UTC
Created attachment 37477 [details]
backstraces_2909
Comment 26 wei-mark.zheng@nokia-sbell.com 2020-09-29 09:29:05 UTC
The backstraces are collected, please check.Thanks.
Comment 27 wei-mark.zheng@nokia-sbell.com 2020-09-30 02:21:26 UTC
Created attachment 37478 [details]
backtraces_0930
Comment 28 Joe Orton 2020-09-30 09:00:34 UTC
It is quite hard to find actual backtraces in that information.

Please attach JUST the backtraces from running "thread apply all bt" from with gdb.

The one I can find in there looks like:

Stack trace of thread 3411716:
#0  0x00007fcf5a0ae744 __pthread_rwlock_wrlock (libpthread.so.0)
#1  0x00007fcf59be1a49 CRYPTO_THREAD_write_lock (libcrypto.so.1.1)
#2  0x00007fcf59ba4b07 RAND_get_rand_method (libcrypto.so.1.1)
#3  0x00007fcf59ba4d82 RAND_seed (libcrypto.so.1.1)
#4  0x00007fcf59da25ee ssl_rand_seed (mod_ssl.so)
#5  0x00007fcf59d8d831 ssl_init_ssl_connection (mod_ssl.so)
#6  0x0000563ebc834eed n/a (/usr/sbin/httpd)
Comment 29 Ruediger Pluem 2020-09-30 11:39:59 UTC
There seem to be further ones:

#0  0x00007fcf591f2664 in __do_global_dtors_aux () from /lib64/libnss_files.so.2
[Current thread is 1 (Thread 0x7fcf59dc0900 (LWP 3563021))]
(gdb) bt
#0  0x00007fcf591f2664 in __do_global_dtors_aux () from /lib64/libnss_files.so.2
#1  0x00007fcf5a30d2eb in _dl_fini () at dl-fini.c:138
#2  0x00007fcf59f0ee87 in __run_exit_handlers (status=status@entry=0, listp=0x7fcf5a092578 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#3  0x00007fcf59f0f040 in __GI_exit (status=status@entry=0) at exit.c:139
#4  0x00007fcf5921d716 in clean_child_exit (code=code@entry=0) at event.c:738
#5  0x00007fcf5921d73d in just_die (sig=<optimized out>) at event.c:743
#6  <signal handler called>
#7  pthread_sigmask (how=how@entry=1, newmask=<optimized out>, newmask@entry=0x7fff02b26230, oldmask=oldmask@entry=0x0)
    at ../sysdeps/unix/sysv/linux/pthread_sigmask.c:48
#8  0x00007fcf5921ccd5 in unblock_signal (sig=sig@entry=15) at event.c:1264
#9  0x00007fcf5921e5d4 in child_main (child_num_arg=child_num_arg@entry=14, child_bucket=child_bucket@entry=0) at event.c:2586
#10 0x00007fcf5921e914 in make_child (s=0x563ebcececf0, slot=slot@entry=14, bucket=bucket@entry=0) at event.c:2691
#11 0x00007fcf5921f290 in perform_idle_server_maintenance (num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:2886
#12 server_main_loop (num_buckets=1, remaining_children_to_start=0) at event.c:3015
#13 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:3092
#14 0x0000563ebc837ce0 in ap_run_mpm (pconf=0x563ebcea5a48, plog=0x563ebced2c68, s=0x563ebcececf0) at mpm_common.c:94
#15 0x0000563ebc821eb3 in main (argc=<optimized out>, argv=<optimized out>) at main.c:819

and

#0  0x00007fcf59e0b5f0 in __do_global_dtors_aux () from /lib64/liblzma.so.5
#1  0x00007fcf5a30d2eb in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#2  0x00007fcf59f0ee87 in __run_exit_handlers () from /lib64/libc.so.6
#3  0x00007fcf59f0f040 in exit () from /lib64/libc.so.6
#4  0x00007fcf5921d716 in clean_child_exit () from /usr/lib64/httpd/modules/mod_mpm_event.so
#5  0x00007fcf5921d73d in just_die () from /usr/lib64/httpd/modules/mod_mpm_event.so
#6  <signal handler called>
#7  0x00007fcf5a0b13cb in pthread_sigmask () from /lib64/libpthread.so.0
#8  0x00007fcf5921ccd5 in unblock_signal () from /usr/lib64/httpd/modules/mod_mpm_event.so
#9  0x00007fcf5921e5d4 in child_main () from /usr/lib64/httpd/modules/mod_mpm_event.so
#10 0x00007fcf5921e914 in make_child () from /usr/lib64/httpd/modules/mod_mpm_event.so
#11 0x00007fcf5921f290 in event_run () from /usr/lib64/httpd/modules/mod_mpm_event.so
#12 0x0000563ebc837ce0 in ap_run_mpm ()
#13 0x0000563ebc821eb3 in main ()

Looks like crashes in library shutdown handlers when httpd is stopped.
Do we ever use liblzma in vanialla httpd?
Comment 30 wei-mark.zheng@nokia-sbell.com 2020-10-09 07:31:56 UTC
Hi,
  Checked from SW team, in our codes we don’t have anything related to lzma.
Comment 31 wei-mark.zheng@nokia-sbell.com 2020-10-09 09:07:43 UTC
Attached here backtarces with “thread apply all bt”also
Comment 32 wei-mark.zheng@nokia-sbell.com 2020-10-09 09:10:07 UTC
Created attachment 37489 [details]
backtraces_0910
Comment 33 Ruediger Pluem 2020-10-09 09:23:11 UTC
Which version of openssl do you use? Is it taken from a distribution package or do you compile it on your own?
Comment 34 wei-mark.zheng@nokia-sbell.com 2020-10-12 05:39:40 UTC
Hi,
  We are currently using openssl-1.1.1g in platform. We will take the same from the distribution and compile it locally for FPLD purpose.
Comment 36 wei-mark.zheng@nokia-sbell.com 2020-10-14 01:52:47 UTC
Hi,
  I am not able to access this link due to permission issue. how can this permission granted ?
http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/event/event.c?r1=1882370&r2=1882369&pathrev=1882370&view=patch
Comment 37 wei-mark.zheng@nokia-sbell.com 2020-10-14 05:57:18 UTC
It was proxy issue, now the link is accessible.
I will share this patch to team and come back later on.
Comment 38 wei-mark.zheng@nokia-sbell.com 2020-10-14 06:26:26 UTC
Hi Ruediger Pluem,
  Any instruction on how to patch it ?
Comment 39 Ruediger Pluem 2020-10-14 13:42:41 UTC
(In reply to wei-mark.zheng@nokia-sbell.com from comment #38)
> Hi Ruediger Pluem,
>   Any instruction on how to patch it ?

Like any patch it is applied to the source code with the patch command (or something similar like svn patch or git apply). For the patch command -p3 seems to be a sensible option when you running this command from the root of the Apache source. Afterwards you follow just your further build steps.
Comment 40 Yann Ylavic 2021-04-18 17:08:46 UTC
Backported to upcoming 2.4 (r1888917).
Will be in the next release.