Bug 63687 - High Memory usage after upgrade to 2.4.41
Summary: High Memory usage after upgrade to 2.4.41
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Core (show other bugs)
Version: 2.4.41
Hardware: PC Linux
: P2 major (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-08-23 06:56 UTC by nitop
Modified: 2019-09-17 12:42 UTC (History)
2 users (show)



Attachments
High Memory usage (85.63 KB, image/png)
2019-08-23 06:57 UTC, nitop
Details
apache2.conf (2.73 KB, text/plain)
2019-08-23 14:29 UTC, nitop
Details
Apache2.4.41 Memory Usage without APR- and pcre-update (157.84 KB, image/png)
2019-08-29 10:15 UTC, nitop
Details

Note You need to log in before you can comment on or make changes to this bug.
Description nitop 2019-08-23 06:56:28 UTC
Hello,

after we upgraded 7 of our systems to apr-1.7.0 (from 1.6.5), pcre-8.43 (from 8.42) and apache 2.4.41 (from 2.4.39) we have a hugh memory problem - you can see this in the screenshot attached. Upgrade was performed on 20 Aug.

We do not changed anything in the config-files.
After rolling back to the old versions, everything is fine.

Any advice?
Comment 1 nitop 2019-08-23 06:57:09 UTC
Created attachment 36732 [details]
High Memory usage

High Memory usage after upgrading
Comment 2 nitop 2019-08-23 07:00:20 UTC
Additional details:
All systems are running with Debian and Kernel "4.4.186".
Comment 3 nitop 2019-08-23 14:29:00 UTC
Created attachment 36733 [details]
apache2.conf

httpd/apache2 config
Comment 4 Joe Orton 2019-08-28 12:22:38 UTC
You've made three changes at the same time which increases the difficult in diagnosing this.  Can you try httpd 2.4.41 on the old APR/PCRE versions, and see if that also has the same memory problem?
Comment 5 Curtis Wilson 2019-08-29 03:42:08 UTC
I have run into the same issue on my servers, they are all Centos 6 and are running cPanel. After the updates I found that I was getting constant issues with memory use from apache in the worker mpm, most times when I would get to them all httpd processes were using 8-12% memory, and causing the boxes to overcommit. 

They updated to 
Apache 2.4.41 (8/22/2019)
APR 1.7 (07/03/2019)
pcre is at 7.8-7

Apache is being obtained from the cPanel easyapache repository.
Comment 6 Curtis Wilson 2019-08-29 03:43:17 UTC
Also to note, until I can figure out the cause as to why this is happening, I have had to downgrade them all to Apache 2.4.39.
Comment 7 Ruediger Pluem 2019-08-29 08:00:18 UTC
(In reply to Curtis Wilson from comment #5)
> I have run into the same issue on my servers, they are all Centos 6 and are
> running cPanel. After the updates I found that I was getting constant issues
> with memory use from apache in the worker mpm, most times when I would get
> to them all httpd processes were using 8-12% memory, and causing the boxes
> to overcommit. 
> 
> They updated to 
> Apache 2.4.41 (8/22/2019)
> APR 1.7 (07/03/2019)
> pcre is at 7.8-7

So only httpd and APR where updated, correct? pcre remained unchanged?

Like with the other reporter, can you just update Apache to isolate the component that causes this?
Comment 8 Ruediger Pluem 2019-08-29 08:07:00 UTC
(In reply to nitop from comment #3)
> Created attachment 36733 [details]
> apache2.conf
> 
> httpd/apache2 config

The given configuration does not show me which MPM you are using. It may be configured in
/etc/apache2/mods-enabled/*.load
/etc/apache2/mods-enabled/*.conf
Which MPM do you use?
Comment 9 nitop 2019-08-29 08:41:04 UTC
Hello,

"Which MPM do you use?"

-> We use worker.

I've now just updated apache2 to 2.4.41 - so far no problems with Memory BUT, I saw this:

# apache2 -V
Server version: Apache/2.4.41 (Unix)
Server loaded:  APR 1.7.0, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1

Why is this different?
We've compiled APR 1.6.5 and NOT 1.7.0.

Thank you!
Comment 10 Ruediger Pluem 2019-08-29 09:40:40 UTC
(In reply to nitop from comment #9)
> Hello,
> 
> "Which MPM do you use?"
> 
> -> We use worker.
> 
> I've now just updated apache2 to 2.4.41 - so far no problems with Memory
> BUT, I saw this:
> 
> # apache2 -V
> Server version: Apache/2.4.41 (Unix)
> Server loaded:  APR 1.7.0, APR-UTIL 1.6.1
> Compiled using: APR 1.6.5, APR-UTIL 1.6.1
> 
> Why is this different?
> We've compiled APR 1.6.5 and NOT 1.7.0.

This is because you complied it against 1.6.5, but when started the httpd process finds 1.7.0 first and hence loads it.
Comment 11 nitop 2019-08-29 10:14:07 UTC
It's definitly Apache 2.4.41. I've updated only Apache2 this morning at ~9:00am - without APR or PCRE:
Please see attached Memory-Graph (mem_usage_ONLY_Apache2_4_41.PNG)
Comment 12 nitop 2019-08-29 10:15:26 UTC
Created attachment 36743 [details]
Apache2.4.41 Memory Usage without APR- and pcre-update
Comment 13 Ruediger Pluem 2019-08-29 15:41:15 UTC
Have you compiled your Apache with debugging symbols?
Are you able to attach to such a memory consuming process with gdb?
If this is the case it would be helpful if you could use the following .gdbinit for your gdb session http://svn.apache.org/viewvc/httpd/httpd/trunk/.gdbinit?revision=1866078&view=co
Once you attached to such a memory consuming process with gdb using the above .gdbinit please execute the following command on gdb side and report back the output:
dump_all_pools
Comment 14 nitop 2019-09-17 06:20:00 UTC
@RuedigerPluem
"Have you compiled your Apache with debugging symbols?"

No. Is this necessary to go on with gdb?

"Are you able to attach to such a memory consuming process with gdb?"

I've not tried it yet. How should I do this?
I can not use your .gdbinit, it outputs some errors - we have an old debian here and gdb 7.0.1.
Comment 15 Ruediger Pluem 2019-09-17 06:42:20 UTC
(In reply to nitop from comment #14)
> @RuedigerPluem
> "Have you compiled your Apache with debugging symbols?"
> 
> No. Is this necessary to go on with gdb?

Yes, you need the debugging symbols to extract the proper information from the process.

> 
> "Are you able to attach to such a memory consuming process with gdb?"
> 
> I've not tried it yet. How should I do this?

http://httpd.apache.org/dev/debugging.html#backtrace

> I can not use your .gdbinit, it outputs some errors - we have an old debian
> here and gdb 7.0.1.

This is bad. The lowest version I tested the .gdbinit with was with 7.2. What are the error messages?
Comment 16 nitop 2019-09-17 07:22:28 UTC
@RuedigerPluem

Here are the errors with your ".gdbinit":
gdb apache2 16606
Reading symbols from /usr/sbin/apache2...done.
Attaching to program: /usr/sbin/apache2, process 16606

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffc77fa000
0x00007fc035f1b303 in ?? ()
Traceback (most recent call last):
  File "<string>", line 78, in <module>
  File "<string>", line 8, in __init__
AttributeError: 'module' object has no attribute 'COMMAND_USER'
/usr/local/src/mni/.gdbinit:548: Error in sourced command file:
Error while executing Python code.

"Yes, you need the debugging symbols to extract the proper information from the process."

-> Did you mean the flag "--enable-maintainer-mode"?
Comment 17 Ruediger Pluem 2019-09-17 08:02:53 UTC
(In reply to nitop from comment #16)
> @RuedigerPluem
> 
> Here are the errors with your ".gdbinit":
> gdb apache2 16606
> Reading symbols from /usr/sbin/apache2...done.
> Attaching to program: /usr/sbin/apache2, process 16606
> 
> warning: no loadable sections found in added symbol-file system-supplied DSO
> at 0x7fffc77fa000
> 0x00007fc035f1b303 in ?? ()
> Traceback (most recent call last):
>   File "<string>", line 78, in <module>
>   File "<string>", line 8, in __init__
> AttributeError: 'module' object has no attribute 'COMMAND_USER'
> /usr/local/src/mni/.gdbinit:548: Error in sourced command file:
> Error while executing Python code.

I cannot fix this. You would need to use a higher version of gdb in this case. Unfortunately the Python code in .gdbinit is essential for debugging your issue.

> 
> "Yes, you need the debugging symbols to extract the proper information from
> the process."
> 
> -> Did you mean the flag "--enable-maintainer-mode"?

That would be one way. Another less strict one is to

export CFLAGS="-Wall -O2 -g"

before you run the configure script.
Comment 18 nitop 2019-09-17 08:52:03 UTC
I am now running gdb 7.2 and tried it again:

/usr/local/bin/gdb apache2 29505 
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/apache2...done.
Attaching to program: /usr/sbin/apache2, process 29505
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /usr/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/lib/libapr-1.so.0...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
....
....
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffd92a92000
0x00007f7bcee85303 in select () from /lib/libc.so.6
.gdbinit:548: Error in sourced command file:
Python scripting is not supported in this copy of GDB.
(gdb) dump_all_pools
Undefined command: "dump_pool_and_children".  Try "help".
Comment 19 nitop 2019-09-17 09:19:36 UTC
We can not continue with debugging because of larger dependencies (gdb, python, ...)

Can someone else debug that with the same problem?
Comment 20 Ruediger Pluem 2019-09-17 10:37:40 UTC
(In reply to nitop from comment #18)
> I am now running gdb 7.2 and tried it again:

> warning: no loadable sections found in added symbol-file system-supplied DSO
> at 0x7ffd92a92000
> 0x00007f7bcee85303 in select () from /lib/libc.so.6
> .gdbinit:548: Error in sourced command file:
> Python scripting is not supported in this copy of GDB.
> (gdb) dump_all_pools
> Undefined command: "dump_pool_and_children".  Try "help".

This gdb does not have Python scripting enabled.
How did you get this gdb 7.2? Did you compile it on your own or did you download it from somewhere? If you compile it on your own you need to specify --with-python to the configure script of gdb. Of cause this requires Python to be available on your system. Not sure how these Python packages are named in Debian.
Comment 21 nitop 2019-09-17 11:04:33 UTC
Someone else should debug this.
We've some hugh dependencies here and can not get >= gdb7.2 with python2.7 to work.
Comment 22 Ruediger Pluem 2019-09-17 12:42:04 UTC
(In reply to nitop from comment #21)
> Someone else should debug this.
> We've some hugh dependencies here and can not get >= gdb7.2 with python2.7
> to work.

Just for the sake of completeness: My GDB 7.2 uses Python 2.6 as this is what my Centos 6 delivers as OS package.