Bug 67379 - Debugging Memory leak in httpd , leads to generate core
Summary: Debugging Memory leak in httpd , leads to generate core
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Core (show other bugs)
Version: 2.4.54
Hardware: PC Linux
: P2 blocker (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-09-13 12:22 UTC by shahbhav1993
Modified: 2023-09-15 11:38 UTC (History)
1 user (show)


Attachments
httpd gdb log (32.71 KB, text/plain)
2023-09-13 12:22 UTC, shahbhav1993 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description shahbhav1993 2023-09-13 12:22:02 UTC 
Created attachment 39020 [details]
httpd gdb log

Hi Team,

Hope doing Great!!

I have installed below modules in httpd and using below version httpd.
-bash# httpd -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 auth_ax_module (shared)
 auth_jwt_module (shared)
 axapi_1_module (shared)
 access_compat_module (shared)
 alias_module (shared)
 authn_core_module (shared)
 authz_core_module (shared)
 authz_user_module (shared)
 dir_module (shared)
 env_module (shared)
 headers_module (shared)
 log_config_module (shared)
 mime_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_shm_module (shared)
 socache_shmcb_module (shared)
 suexec_module (shared)
 unixd_module (shared)
 mpm_prefork_module (shared)
 ssl_module (shared)
 web_apis_module (shared)
 cgid_module (shared)
 unique_id_module (shared)

-bash# httpd -V
Server version: Apache/2.4.54 (Unix)
Server built:   Jun 30 2022 13:43:50
Server's Module Magic Number: 20120211:124
Server loaded:  APR 1.7.4, APR-UTIL 1.6.3, PCRE 8.35 2014-04-04
Compiled using: APR 1.5.2, APR-UTIL 1.5.2, PCRE 8.32 2012-11-30
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/usr"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/httpd/conf/mime.types"
 -D SERVER_CONFIG_FILE="/etc/httpd/conf/httpd.conf"

I am running below script to restart the httpd ,I am not even browsing web-pages ,after four to five days it keep on generating core. and cpu reaches 100%.

#!/bin/bash

# Define the number of restarts
total_restarts=4000

for ((i=1; i<=$total_restarts; i++)); do
httpd -k restart

    # Wait for a few seconds before the next iteration
    sleep 120
done

I can see /proc/parent_httpd_pid/statm output keep on increasing below I have attached sample output.
2023-08-25 10:45:02 179536 7679 5872 158 0 15097 0
2023-08-25 10:45:30 192359 7712 5901 158 0 15097 0
2023-08-25 10:50:30 218033 7723 5906 158 0 15125 0
...
2023-08-29 03:18:21 33181755 69652 59222 158 0 23737 0
2023-08-29 03:23:21 33220220 69689 59248 158 0 23733 0
2023-08-29 03:28:21 33245866 69769 59322 158 0 23733 0
2023-08-29 03:33:21 33284361 69813 59356 158 0 23759 0

I have tried to attach gdb. but not getting useful information. dump_all_pools command not working for me. what to do for that.
I have attached logs for gdb.
(gdb) dump_all_pools
Undefined command: "dump_all_pools".  Try "help".
(gdb) c

Please let me know if any other information needed from my side.

Thanks,
Bhav
Comment 1 Yann Ylavic 2023-09-14 12:02:33 UTC 
(In reply to shahbhav1993 from comment #0)

> -bash# httpd -M
> Loaded Modules:
> ...
> auth_ax_module (shared)
> auth_jwt_module (shared)
> axapi_1_module (shared)
> ...
>  web_apis_module (shared)
> ...

Did you try without these (third-party) modules?


> -bash# httpd -V
> Server version: Apache/2.4.54 (Unix)
> Server built:   Jun 30 2022 13:43:50
> Server's Module Magic Number: 20120211:124
> Server loaded:  APR 1.7.4, APR-UTIL 1.6.3, PCRE 8.35 2014-04-04
> Compiled using: APR 1.5.2, APR-UTIL 1.5.2, PCRE 8.32 2012-11-30
> Architecture:   64-bit
> Server MPM:     prefork
>   threaded:     no
>     forked:     yes (variable process count)

This says prefork but gdb traces seem to indicate MPM event (explicit LoadModule?), which one is used exactly?


> I can see /proc/parent_httpd_pid/statm output keep on increasing below I
> have attached sample output.
> 2023-08-25 10:45:02 179536 7679 5872 158 0 15097 0
> 2023-08-25 10:45:30 192359 7712 5901 158 0 15097 0
> 2023-08-25 10:50:30 218033 7723 5906 158 0 15125 0
> ...
> 2023-08-29 03:18:21 33181755 69652 59222 158 0 23737 0
> 2023-08-29 03:23:21 33220220 69689 59248 158 0 23733 0
> 2023-08-29 03:28:21 33245866 69769 59322 158 0 23733 0
> 2023-08-29 03:33:21 33284361 69813 59356 158 0 23759 0

This is pretty stable for me with both MPM prefork and event, but without the third-party modules.

> I have tried to attach gdb. but not getting useful information.
> dump_all_pools command not working for me. what to do for that.
> I have attached logs for gdb.
> (gdb) dump_all_pools
> Undefined command: "dump_all_pools".  Try "help".

You probably need the .gdbinit file provided by httpd (https://github.com/apache/httpd/blob/trunk/.gdbinit) in your gdb environment or working directory.
Comment 2 shahbhav1993 2023-09-15 11:38:03 UTC 
(In reply to Yann Ylavic from comment #1)
> (In reply to shahbhav1993 from comment #0)
> 
> > -bash# httpd -M
> > Loaded Modules:
> > ...
> > auth_ax_module (shared)
> > auth_jwt_module (shared)
> > axapi_1_module (shared)
> > ...
> >  web_apis_module (shared)
> > ...
> 
> Did you try without these (third-party) modules?
I didn't try. I will try to remove these 4 modules, which you highlighted.
Just to confirm other modules other than these 4, are supported by apache?
> 
> 
> > -bash# httpd -V
> > Server version: Apache/2.4.54 (Unix)
> > Server built:   Jun 30 2022 13:43:50
> > Server's Module Magic Number: 20120211:124
> > Server loaded:  APR 1.7.4, APR-UTIL 1.6.3, PCRE 8.35 2014-04-04
> > Compiled using: APR 1.5.2, APR-UTIL 1.5.2, PCRE 8.32 2012-11-30
> > Architecture:   64-bit
> > Server MPM:     prefork
> >   threaded:     no
> >     forked:     yes (variable process count)
> 
> This says prefork but gdb traces seem to indicate MPM event (explicit
> LoadModule?), which one is used exactly?

I am using prefork. here is log from gdb. 
oaded symbols for /etc/httpd/modules/mod_cgid.so
Reading symbols from /etc/httpd/modules/mod_unique_id.so...Missing separate debuginfo for /etc/httpd/modules/mod_unique_id.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/44/d70e82fd0afc88216182b8840f2965753b745d.debug
Reading symbols from /etc/httpd/modules/mod_unique_id.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_unique_id.so
0x00007f4fe7c2eb23 in __select_nocancel () from /lib64/libc.so.6
(gdb) c
Continuing.

Program received signal SIGHUP, Hangup.
0x00007f4fe7c2eb23 in __select_nocancel () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f4fe7c2eb23 in __select_nocancel () at /lib64/libc.so.6
#1  0x00007f4fe83558a5 in apr_sleep () at /lib64/libapr-1.so.0
#2  0x000055f5eb44a059 in ap_wait_or_timeout ()
#3  0x00007f3526282682 in prefork_run () at /etc/httpd/modules/mod_mpm_prefork.so
#4  0x000055f5eb4494ae in ap_run_mpm ()
#5  0x000055f5eb44127d in main ()
(gdb) c
Continuing.

Program received signal SIGHUP, Hangup.
0x00007f4fe7b6f657 in kill () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f4fe7b6f657 in kill () at /lib64/libc.so.6
#1  0x00007f3526282dc1 in prefork_run () at /etc/httpd/modules/mod_mpm_prefork.so
#2  0x000055f5eb4494ae in ap_run_mpm ()
#3  0x000055f5eb44127d in main ()
(gdb) c
Continuing.

Program received signal SIGPIPE, Broken pipe.
0x00007f4fe81196e0 in __write_nocancel () from /lib64/libpthread.so.0
(gdb) bt full
#0  0x00007f4fe81196e0 in __write_nocancel () at /lib64/libpthread.so.0
#1  0x00007f4fe8345e4c in apr_file_write () at /lib64/libapr-1.so.0
#2  0x00007f4fe834462e in apr_file_write_full () at /lib64/libapr-1.so.0
#3  0x000055f5eb46e4c4 in log_error_core ()
#4  0x000055f5eb46ed78 in ap_log_error_ ()
#5  0x00007f3526282e0a in prefork_run () at /etc/httpd/modules/mod_mpm_prefork.so
#6  0x000055f5eb4494ae in ap_run_mpm ()
#7  0x000055f5eb44127d in main ()
(gdb) c
Continuing.
warning: Could not load shared library symbols for 4 libraries, e.g. ../lib/libnetsnmpagent.so.15.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
[Detaching after fork from child process 28341]
[Detaching after fork from child process 28345]
[Detaching after fork from child process 28350]
[Detaching after fork from child process 28354]
[Detaching after fork from child process 28365]
[Detaching after fork from child process 28366]
[Detaching after fork from child process 28367]
[Detaching after fork from child process 28368]
[Detaching after fork from child process 28369]
[Detaching after fork from child process 28370]

Program received signal SIGINT, Interrupt.
0x00007f4fe7c2eb23 in __select_nocancel () from /lib64/libc.so.6
(gdb) bt full
#0  0x00007f4fe7c2eb23 in __select_nocancel () at /lib64/libc.so.6
#1  0x00007f4fe83558a5 in apr_sleep () at /lib64/libapr-1.so.0
#2  0x000055f5eb44a059 in ap_wait_or_timeout ()
#3  0x00007f3520651682 in prefork_run () at /etc/httpd/modules/mod_mpm_prefork.so
#4  0x000055f5eb4494ae in ap_run_mpm ()
#5  0x000055f5eb44127d in main ()
(gdb)

> 
> 
> > I can see /proc/parent_httpd_pid/statm output keep on increasing below I
> > have attached sample output.
> > 2023-08-25 10:45:02 179536 7679 5872 158 0 15097 0
> > 2023-08-25 10:45:30 192359 7712 5901 158 0 15097 0
> > 2023-08-25 10:50:30 218033 7723 5906 158 0 15125 0
> > ...
> > 2023-08-29 03:18:21 33181755 69652 59222 158 0 23737 0
> > 2023-08-29 03:23:21 33220220 69689 59248 158 0 23733 0
> > 2023-08-29 03:28:21 33245866 69769 59322 158 0 23733 0
> > 2023-08-29 03:33:21 33284361 69813 59356 158 0 23759 0
> 
> This is pretty stable for me with both MPM prefork and event, but without
> the third-party modules.

this output increase for me when , I do httpd -k restart every 3 min interval, every restart it increase gradually. are you doing restart and includes other shared modules without third-party modules?
 do you mean without these 4 modules auth_ax_module (shared) ,auth_jwt_module (shared), axapi_1_module (shared), web_apis_module (shared) you tried?
If so I will try without these four module.

> > I have tried to attach gdb. but not getting useful information.
> > dump_all_pools command not working for me. what to do for that.
> > I have attached logs for gdb.
> > (gdb) dump_all_pools
> > Undefined command: "dump_all_pools".  Try "help".
> 
> You probably need the .gdbinit file provided by httpd
> (https://github.com/apache/httpd/blob/trunk/.gdbinit) in your gdb
> environment or working directory.

how to include this while doing gdb? 
I usually do "gdb -p  pid"

Thanks & Regards,
Bhav