Bug 64533

Created attachment 37314 [details]
coredump

Created attachment 37315 [details]
coredump_2

We need:

1. Stacktraces from gdb (see http://httpd.apache.org/dev/debugging.html#crashes). The coredumps need to be analyzed on the system where they got created.
2. We need the error and access logs that were recorded during the crash. The more verbose the error logs the better.
3. We need the configuration used during the test.

Created attachment 37323 [details]
backtrace

Created attachment 37324 [details]
configuration files

The backtrace and configuration files attached for your further checking. Please let us know if anything needed.

Unfortunately the stacktraces do not help as they are not complete. Please try to install debugging symbols for APR / APR-UTIL and httpd as well.

Created attachment 37355 [details]
debug logs with 2.4.43

we have reproduced the issue and collected the logs again with http 2.4.43.
Attached the tar file which has details. Please check.

Unfortunately the stacktraces are still incomplete.

Thanks for the feedback.
We are not sure what step is missing during the stacktraces capture.
Could you please kindly give some guidelines on the procedure to capture a full stacktraces for your analysis ? Thanks.

BTW: could you share your mailaddress, then it would be more effective to discuss directly in the mail.

Created attachment 37358 [details]
backtraces_0710

Latest backtraces attached. This is following the guidelines http://httpd.apache.org/dev/debugging.html#crashes. If this is  no completed traces, then can we have a virtual-meeting or mail discussion to discuss how to proceeded for us ? Thanks.

any updates on this case ?

Unfortunateley the backtraces are inclomplete and do not tell us where the crash happens. I don't know why the instructions on http://httpd.apache.org/dev/debugging.html#crashes do not deliver correct backtraces. It is likely related to your Linux setup. Which Linux distro are you using?

Hi Ruediger Pluem,
   The linux distro we are using is linux MIPS64, kernel version 4.4.227.

[root@CFPU-0(RNC-1009) /root]
 # uname -ar
Linux CFPU-0 4.4.227-octeon-distro.git-v2.105-2-rc-wnd #1 SMP Fri Jul 31 11:39:15 UTC 2020 mips64 mips64 mips64 GNU/Linux

Are you able to reproduce the same issue with a different Linux distribution?

Yes, we have observed the crash starting from the Apache/2.4.41 which is based on another old kernel version.  
But this issue was not observed from old Apache version until Apache/2.4.39.

Hi Ruediger Pluem,
  Can you share any debug options which can be used for compiling httpd which can further be used to get more information ? 
  Also could it possible for you to have remote debugging session where you can login to our environment directly ? Thanks.

As said, you should try to reproduce the issue on a more common Linux distribution (Debian or RedHat based) and probably on non MIPS hardware. It looks like your current setup is not capable of producing the debug information needed for further investigations.

Hi Ruediger Pluem,
   I understood your suggestion. Here the situation is that our product is based on MIPS hardware and linux distribution for quite long time. And we are using apache component from the start of our product when it works fine without issue.
The problem occurs starting from version 2.4.41 and remains still now. So it is likely the problem was introduced in version 2.4.41 by new software code.
  Some questions:
1. Do you receive similar issue from other products using same apache version ?
2. Is there any other possible way to help debug this issue ?
  Please kindly suggest, thanks !

(In reply to wei-mark.zheng@nokia-sbell.com from comment #21)
> Hi Ruediger Pluem,
>    I understood your suggestion. Here the situation is that our product is
> based on MIPS hardware and linux distribution for quite long time. And we
> are using apache component from the start of our product when it works fine
> without issue.
> The problem occurs starting from version 2.4.41 and remains still now. So it
> is likely the problem was introduced in version 2.4.41 by new software code.
>   Some questions:
> 1. Do you receive similar issue from other products using same apache
> version ?

No.

> 2. Is there any other possible way to help debug this issue ?
>   Please kindly suggest, thanks !

I am out of ideas for your platform. Hence the idea of you doing your fuzzing against a more common platform where we could possibly get the data we need if the issue is reproducable there.

We have observed the crash for new 2.4.46 version as well.
Server version: Apache/2.4.46 (Fedora) Server

In addition, the crash is seen when https [Port 80] and TLS [port 443] codenomicon suites are run together which sending malformed packets continously.
But no crash observed when both the suites are run separately.

(In reply to wei-mark.zheng@nokia-sbell.com from comment #23)
> We have observed the crash for new 2.4.46 version as well.
> Server version: Apache/2.4.46 (Fedora) Server

Looks like you are now running your fuzzing tests against a Fedora build and not your Linux MIPS distro. If this is the case can you please provide stacktraces again. Hopefully they are usable then.

Created attachment 37477 [details]
backstraces_2909

The backstraces are collected, please check.Thanks.

Created attachment 37478 [details]
backtraces_0930

It is quite hard to find actual backtraces in that information.

Please attach JUST the backtraces from running "thread apply all bt" from with gdb.

The one I can find in there looks like:

Stack trace of thread 3411716:
#0  0x00007fcf5a0ae744 __pthread_rwlock_wrlock (libpthread.so.0)
#1  0x00007fcf59be1a49 CRYPTO_THREAD_write_lock (libcrypto.so.1.1)
#2  0x00007fcf59ba4b07 RAND_get_rand_method (libcrypto.so.1.1)
#3  0x00007fcf59ba4d82 RAND_seed (libcrypto.so.1.1)
#4  0x00007fcf59da25ee ssl_rand_seed (mod_ssl.so)
#5  0x00007fcf59d8d831 ssl_init_ssl_connection (mod_ssl.so)
#6  0x0000563ebc834eed n/a (/usr/sbin/httpd)

There seem to be further ones:

#0  0x00007fcf591f2664 in __do_global_dtors_aux () from /lib64/libnss_files.so.2
[Current thread is 1 (Thread 0x7fcf59dc0900 (LWP 3563021))]
(gdb) bt
#0  0x00007fcf591f2664 in __do_global_dtors_aux () from /lib64/libnss_files.so.2
#1  0x00007fcf5a30d2eb in _dl_fini () at dl-fini.c:138
#2  0x00007fcf59f0ee87 in __run_exit_handlers (status=status@entry=0, listp=0x7fcf5a092578 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#3  0x00007fcf59f0f040 in __GI_exit (status=status@entry=0) at exit.c:139
#4  0x00007fcf5921d716 in clean_child_exit (code=code@entry=0) at event.c:738
#5  0x00007fcf5921d73d in just_die (sig=<optimized out>) at event.c:743
#6  <signal handler called>
#7  pthread_sigmask (how=how@entry=1, newmask=<optimized out>, newmask@entry=0x7fff02b26230, oldmask=oldmask@entry=0x0)
    at ../sysdeps/unix/sysv/linux/pthread_sigmask.c:48
#8  0x00007fcf5921ccd5 in unblock_signal (sig=sig@entry=15) at event.c:1264
#9  0x00007fcf5921e5d4 in child_main (child_num_arg=child_num_arg@entry=14, child_bucket=child_bucket@entry=0) at event.c:2586
#10 0x00007fcf5921e914 in make_child (s=0x563ebcececf0, slot=slot@entry=14, bucket=bucket@entry=0) at event.c:2691
#11 0x00007fcf5921f290 in perform_idle_server_maintenance (num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:2886
#12 server_main_loop (num_buckets=1, remaining_children_to_start=0) at event.c:3015
#13 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:3092
#14 0x0000563ebc837ce0 in ap_run_mpm (pconf=0x563ebcea5a48, plog=0x563ebced2c68, s=0x563ebcececf0) at mpm_common.c:94
#15 0x0000563ebc821eb3 in main (argc=<optimized out>, argv=<optimized out>) at main.c:819

and

#0  0x00007fcf59e0b5f0 in __do_global_dtors_aux () from /lib64/liblzma.so.5
#1  0x00007fcf5a30d2eb in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#2  0x00007fcf59f0ee87 in __run_exit_handlers () from /lib64/libc.so.6
#3  0x00007fcf59f0f040 in exit () from /lib64/libc.so.6
#4  0x00007fcf5921d716 in clean_child_exit () from /usr/lib64/httpd/modules/mod_mpm_event.so
#5  0x00007fcf5921d73d in just_die () from /usr/lib64/httpd/modules/mod_mpm_event.so
#6  <signal handler called>
#7  0x00007fcf5a0b13cb in pthread_sigmask () from /lib64/libpthread.so.0
#8  0x00007fcf5921ccd5 in unblock_signal () from /usr/lib64/httpd/modules/mod_mpm_event.so
#9  0x00007fcf5921e5d4 in child_main () from /usr/lib64/httpd/modules/mod_mpm_event.so
#10 0x00007fcf5921e914 in make_child () from /usr/lib64/httpd/modules/mod_mpm_event.so
#11 0x00007fcf5921f290 in event_run () from /usr/lib64/httpd/modules/mod_mpm_event.so
#12 0x0000563ebc837ce0 in ap_run_mpm ()
#13 0x0000563ebc821eb3 in main ()

Looks like crashes in library shutdown handlers when httpd is stopped.
Do we ever use liblzma in vanialla httpd?

Hi,
  Checked from SW team, in our codes we don’t have anything related to lzma.

Attached here backtarces with “thread apply all bt”also

Created attachment 37489 [details]
backtraces_0910

Which version of openssl do you use? Is it taken from a distribution package or do you compile it on your own?

Hi,
  We are currently using openssl-1.1.1g in platform. We will take the same from the distribution and compile it locally for FPLD purpose.

Can you please try the patch from r1882370 : http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/event/event.c?r1=1882370&r2=1882369&pathrev=1882370&view=patch

Hi,
  I am not able to access this link due to permission issue. how can this permission granted ?
http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/event/event.c?r1=1882370&r2=1882369&pathrev=1882370&view=patch

It was proxy issue, now the link is accessible.
I will share this patch to team and come back later on.

Hi Ruediger Pluem,
  Any instruction on how to patch it ?

(In reply to wei-mark.zheng@nokia-sbell.com from comment #38)
> Hi Ruediger Pluem,
>   Any instruction on how to patch it ?

Like any patch it is applied to the source code with the patch command (or something similar like svn patch or git apply). For the patch command -p3 seems to be a sensible option when you running this command from the root of the Apache source. Afterwards you follow just your further build steps.

Backported to upcoming 2.4 (r1888917).
Will be in the next release.