Hello, after we upgraded 7 of our systems to apr-1.7.0 (from 1.6.5), pcre-8.43 (from 8.42) and apache 2.4.41 (from 2.4.39) we have a hugh memory problem - you can see this in the screenshot attached. Upgrade was performed on 20 Aug. We do not changed anything in the config-files. After rolling back to the old versions, everything is fine. Any advice?
Created attachment 36732 [details] High Memory usage High Memory usage after upgrading
Additional details: All systems are running with Debian and Kernel "4.4.186".
Created attachment 36733 [details] apache2.conf httpd/apache2 config
You've made three changes at the same time which increases the difficult in diagnosing this. Can you try httpd 2.4.41 on the old APR/PCRE versions, and see if that also has the same memory problem?
I have run into the same issue on my servers, they are all Centos 6 and are running cPanel. After the updates I found that I was getting constant issues with memory use from apache in the worker mpm, most times when I would get to them all httpd processes were using 8-12% memory, and causing the boxes to overcommit. They updated to Apache 2.4.41 (8/22/2019) APR 1.7 (07/03/2019) pcre is at 7.8-7 Apache is being obtained from the cPanel easyapache repository.
Also to note, until I can figure out the cause as to why this is happening, I have had to downgrade them all to Apache 2.4.39.
(In reply to Curtis Wilson from comment #5) > I have run into the same issue on my servers, they are all Centos 6 and are > running cPanel. After the updates I found that I was getting constant issues > with memory use from apache in the worker mpm, most times when I would get > to them all httpd processes were using 8-12% memory, and causing the boxes > to overcommit. > > They updated to > Apache 2.4.41 (8/22/2019) > APR 1.7 (07/03/2019) > pcre is at 7.8-7 So only httpd and APR where updated, correct? pcre remained unchanged? Like with the other reporter, can you just update Apache to isolate the component that causes this?
(In reply to nitop from comment #3) > Created attachment 36733 [details] > apache2.conf > > httpd/apache2 config The given configuration does not show me which MPM you are using. It may be configured in /etc/apache2/mods-enabled/*.load /etc/apache2/mods-enabled/*.conf Which MPM do you use?
Hello, "Which MPM do you use?" -> We use worker. I've now just updated apache2 to 2.4.41 - so far no problems with Memory BUT, I saw this: # apache2 -V Server version: Apache/2.4.41 (Unix) Server loaded: APR 1.7.0, APR-UTIL 1.6.1 Compiled using: APR 1.6.5, APR-UTIL 1.6.1 Why is this different? We've compiled APR 1.6.5 and NOT 1.7.0. Thank you!
(In reply to nitop from comment #9) > Hello, > > "Which MPM do you use?" > > -> We use worker. > > I've now just updated apache2 to 2.4.41 - so far no problems with Memory > BUT, I saw this: > > # apache2 -V > Server version: Apache/2.4.41 (Unix) > Server loaded: APR 1.7.0, APR-UTIL 1.6.1 > Compiled using: APR 1.6.5, APR-UTIL 1.6.1 > > Why is this different? > We've compiled APR 1.6.5 and NOT 1.7.0. This is because you complied it against 1.6.5, but when started the httpd process finds 1.7.0 first and hence loads it.
It's definitly Apache 2.4.41. I've updated only Apache2 this morning at ~9:00am - without APR or PCRE: Please see attached Memory-Graph (mem_usage_ONLY_Apache2_4_41.PNG)
Created attachment 36743 [details] Apache2.4.41 Memory Usage without APR- and pcre-update
Have you compiled your Apache with debugging symbols? Are you able to attach to such a memory consuming process with gdb? If this is the case it would be helpful if you could use the following .gdbinit for your gdb session http://svn.apache.org/viewvc/httpd/httpd/trunk/.gdbinit?revision=1866078&view=co Once you attached to such a memory consuming process with gdb using the above .gdbinit please execute the following command on gdb side and report back the output: dump_all_pools
@RuedigerPluem "Have you compiled your Apache with debugging symbols?" No. Is this necessary to go on with gdb? "Are you able to attach to such a memory consuming process with gdb?" I've not tried it yet. How should I do this? I can not use your .gdbinit, it outputs some errors - we have an old debian here and gdb 7.0.1.
(In reply to nitop from comment #14) > @RuedigerPluem > "Have you compiled your Apache with debugging symbols?" > > No. Is this necessary to go on with gdb? Yes, you need the debugging symbols to extract the proper information from the process. > > "Are you able to attach to such a memory consuming process with gdb?" > > I've not tried it yet. How should I do this? http://httpd.apache.org/dev/debugging.html#backtrace > I can not use your .gdbinit, it outputs some errors - we have an old debian > here and gdb 7.0.1. This is bad. The lowest version I tested the .gdbinit with was with 7.2. What are the error messages?
@RuedigerPluem Here are the errors with your ".gdbinit": gdb apache2 16606 Reading symbols from /usr/sbin/apache2...done. Attaching to program: /usr/sbin/apache2, process 16606 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffc77fa000 0x00007fc035f1b303 in ?? () Traceback (most recent call last): File "<string>", line 78, in <module> File "<string>", line 8, in __init__ AttributeError: 'module' object has no attribute 'COMMAND_USER' /usr/local/src/mni/.gdbinit:548: Error in sourced command file: Error while executing Python code. "Yes, you need the debugging symbols to extract the proper information from the process." -> Did you mean the flag "--enable-maintainer-mode"?
(In reply to nitop from comment #16) > @RuedigerPluem > > Here are the errors with your ".gdbinit": > gdb apache2 16606 > Reading symbols from /usr/sbin/apache2...done. > Attaching to program: /usr/sbin/apache2, process 16606 > > warning: no loadable sections found in added symbol-file system-supplied DSO > at 0x7fffc77fa000 > 0x00007fc035f1b303 in ?? () > Traceback (most recent call last): > File "<string>", line 78, in <module> > File "<string>", line 8, in __init__ > AttributeError: 'module' object has no attribute 'COMMAND_USER' > /usr/local/src/mni/.gdbinit:548: Error in sourced command file: > Error while executing Python code. I cannot fix this. You would need to use a higher version of gdb in this case. Unfortunately the Python code in .gdbinit is essential for debugging your issue. > > "Yes, you need the debugging symbols to extract the proper information from > the process." > > -> Did you mean the flag "--enable-maintainer-mode"? That would be one way. Another less strict one is to export CFLAGS="-Wall -O2 -g" before you run the configure script.
I am now running gdb 7.2 and tried it again: /usr/local/bin/gdb apache2 29505 GNU gdb (GDB) 7.2 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/apache2...done. Attaching to program: /usr/sbin/apache2, process 29505 Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.3 Reading symbols from /usr/lib/libaprutil-1.so.0...done. Loaded symbols for /usr/lib/libaprutil-1.so.0 Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /usr/lib/libapr-1.so.0...done. Loaded symbols for /usr/lib/libapr-1.so.0 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_nis.so.2 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 .... .... warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffd92a92000 0x00007f7bcee85303 in select () from /lib/libc.so.6 .gdbinit:548: Error in sourced command file: Python scripting is not supported in this copy of GDB. (gdb) dump_all_pools Undefined command: "dump_pool_and_children". Try "help".
We can not continue with debugging because of larger dependencies (gdb, python, ...) Can someone else debug that with the same problem?
(In reply to nitop from comment #18) > I am now running gdb 7.2 and tried it again: > warning: no loadable sections found in added symbol-file system-supplied DSO > at 0x7ffd92a92000 > 0x00007f7bcee85303 in select () from /lib/libc.so.6 > .gdbinit:548: Error in sourced command file: > Python scripting is not supported in this copy of GDB. > (gdb) dump_all_pools > Undefined command: "dump_pool_and_children". Try "help". This gdb does not have Python scripting enabled. How did you get this gdb 7.2? Did you compile it on your own or did you download it from somewhere? If you compile it on your own you need to specify --with-python to the configure script of gdb. Of cause this requires Python to be available on your system. Not sure how these Python packages are named in Debian.
Someone else should debug this. We've some hugh dependencies here and can not get >= gdb7.2 with python2.7 to work.
(In reply to nitop from comment #21) > Someone else should debug this. > We've some hugh dependencies here and can not get >= gdb7.2 with python2.7 > to work. Just for the sake of completeness: My GDB 7.2 uses Python 2.6 as this is what my Centos 6 delivers as OS package.
I am wondering if the changes to PCRE_DOTALL is doing this. If using regex a lot try: RegexDefaultOptions -DOTALL
Are you using HTTP2 and/or mod_md at all? This will allow us to pare down the diffs
@Jim Jagielski We do not use http2 or mod_md.
Thanks... that helps to narrow down things quite a bit.
(In reply to Ruediger Pluem from comment #7) > (In reply to Curtis Wilson from comment #5) > > I have run into the same issue on my servers, they are all Centos 6 and are > > running cPanel. After the updates I found that I was getting constant issues > > with memory use from apache in the worker mpm, most times when I would get > > to them all httpd processes were using 8-12% memory, and causing the boxes > > to overcommit. > > > > They updated to > > Apache 2.4.41 (8/22/2019) > > APR 1.7 (07/03/2019) > > pcre is at 7.8-7 > > So only httpd and APR where updated, correct? pcre remained unchanged? > > Like with the other reporter, can you just update Apache to isolate the > component that causes this? Hy Ruediger, So, I was with the same problem here and i found this topic talking exactly about. And the problem is aparently solved. I would like to explain the situation in hope this help you to identify another causes like this. Here comes the history.. My server is an Amazon Linux. The problematic httpd 2.4.41 has been compiled from source because a security audit demanded. In that day I could not install it through yum because the amazon linux repositories already had the last apache version, i don´t remember exactly, but think the 2.4.34 and the security reports suggested the 2.4.41. Ok this was the history. So in your Comments #5 and #7 I was about to follow that you suggested but, before I would like to test if my httpd's earlier version would work normally, and i did. I renamed my current apache directory, undo the custom settings and installed the "previous" apache version throught yum. To my great surprise yum told me it would install 2.4.41 version. I got a little confused but if yum gave me 2.4.41, that´s ok. Ahh, I forgot to talk that the MPM chosen was the Event. Imediately after install i check the version with httpd -V (After a few corrections to make sure the httpd binary's running was that installed by yum) and the result was as follow: My Server with Memory Leakage: Server version: Apache/2.4.41 (Unix) Server built: Nov 6 2019 00:42:00 Server's Module Magic Number: 20120211:88 Server loaded: APR 1.7.0, APR-UTIL 1.6.1 Compiled using: APR 1.7.0, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) The same server with memory leakage fixed after install httpd 2.4.41 throught yum (Actally not the same server, the above server it´s a clone in homologation enviroment): Server version: Apache/2.4.41 () Server built: Oct 22 2019 22:59:04 Server's Module Magic Number: 20120211:88 Server loaded: APR 1.6.3, APR-UTIL 1.6.1 Compiled using: APR 1.6.3, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) Another thing maybe help you is that these two servers above were replicated from another server's image, also running apache. This image has been used to create about 10 servers and none of them have this problem. Here is the httpd -V result from the root server: Server version: Apache/2.4.34 () Server built: Aug 17 2018 22:14:33 Server's Module Magic Number: 20120211:79 Server loaded: APR 1.6.3, APR-UTIL 1.6.1 Compiled using: APR 1.6.3, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) Ok, as i see, the problematic guy was compiled using APR 1.7.0 and all of the others using APR 1.6.3. I didn´t did none of your suggested tests. Until now, my problem aparently resolved, as 1 day has been passed and no memory leaks. In fact, memory consumption didn´t even go up. My server has 2 GB RAM, it runs just one application, with just one connection for just one user, and with memory leakage, the server was freezing after about 5 hours of use. The server with no problem, runs 5 httpd children, and each one consumes about 27MB each. The problematic starts with 3 children and the memory grows up to ceiling.
It still doesn't look good here. I've compiled "APR 1.6.3" and "APR-UTIL 1.6.1" with "Apache/2.4.41". APR 1.6.3: ./configure --prefix=/usr/local/apr/ make make install APR-UTIL 1.6.1: ./configure --prefix=/usr/local/apr/ --with-apr=/usr/local/apr/ make make install Apache2: --with-apr=/usr/local/apr/bin/apr-1-config --with-apr-util=/usr/local/apr/bin/apu-1-config --> High Memory usage again.
apache2ctl -V Server version: Apache/2.4.41 (Unix) Server built: Dec 9 2019 08:44:12 Server's Module Magic Number: 20120211:88 Server loaded: APR 1.6.3, APR-UTIL 1.6.1 Compiled using: APR 1.6.3, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: worker threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="" -D SUEXEC_BIN="/usr/lib/apache2/suexec" -D DEFAULT_PIDLOG="/var/run/apache2/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types" -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"