Bug 51006 - Apache consume all memory and swap when setup as reverse proxy
Summary: Apache consume all memory and swap when setup as reverse proxy
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.2.17
Hardware: HP Linux
: P2 blocker (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate
Depends on:
Blocks:
 
Reported: 2011-04-01 09:31 UTC by Martin Cmelik
Modified: 2018-11-07 21:09 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Cmelik 2011-04-01 09:31:14 UTC
Hi,

for a long time/days I have try to upgrade our current reverse proxies farm from version 2.0.61 to 2.2.17.

With default config it start without problem, but when I include our config of reverse proxies it consume all memory (3GB) and after that whole swap (2GB) on machine where same config works well on 2.0.61

In our config is 14k virtual hosts as ProxyPass configured in this way:

<VirtualHost *>
ServerName test.example.com
ServerAdmin postmaster@example.com
ProxyPass / http://1.1.1.1:1234/
ProxyPassReverse / http://1.1.1.1:1234/
ProxyPreserveHost On
</VirtualHost>
...

I have take config.nice from old one and compile 2.2.17 in same way:

[root@redhat-testing opt]# cat /usr/src/httpd-2.2.17/config.nice
#! /bin/sh
#
# Created by configure

"./configure" \
"--prefix=/opt/apache" \
"--exec-prefix=/opt/apache" \
"--enable-cache" \
"--enable-disk-cache" \
"--enable-mem-cache" \
"--enable-logio" \
"--enable-headers" \
"--enable-usertrack" \
"--enable-version" \
"--enable-proxy" \
"--enable-proxy-connect" \
"--enable-proxy-ftp" \
"--enable-proxy-http" \
"--enable-static-support" \
"--enable-static-htpasswd" \
"--enable-static-htdigest" \
"--enable-static-rotatelogs" \
"--enable-static-logresolve" \
"--enable-static-htdbm" \
"--enable-static-ab" \
"--enable-static-checkgid" \
"--enable-http" \
"--enable-info" \
"--enable-cgi" \
"--enable-vhost-alias" \
"--disable-userdir" \
"--enable-rewrite" \
"--enable-so" \
"--with-mpm=prefork" \
"$@"

#./config.nice
#make
#make install

[root@redhat-testing opt]# /opt/apache/bin/apachectl -V
Server version: Apache/2.2.17 (Unix)
Server built:   Apr  1 2011 08:41:55
Server's Module Magic Number: 20051115:25
Server loaded:  APR 1.4.2, APR-Util 1.3.10
Compiled using: APR 1.4.2, APR-Util 1.3.10
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/apache"
 -D SUEXEC_BIN="/opt/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

[root@redhat-testing opt]# /opt/apache/bin/apachectl -l
Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_cache.c
  mod_disk_cache.c
  mod_mem_cache.c
  mod_include.c
  mod_filter.c
  mod_log_config.c
  mod_logio.c
  mod_env.c
  mod_headers.c
  mod_usertrack.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_scgi.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_info.c
  mod_cgi.c
  mod_vhost_alias.c
  mod_negotiation.c
  mod_dir.c
  mod_actions.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c

Prefork configuration:

<IfModule prefork.c>
StartServers         100
MinSpareServers      50
MaxSpareServers     100
ServerLimit        1600
MaxClients         1600
MaxRequestsPerChild  25000
</IfModule>


In normal state (2.0.61) when you start apache it will create one root server process and waiting for connection (then create childs).
But on 2.2.17 it will start up all StartServers and Childs as well and in next 15 seconds start swaping until consume all of available space.

When I setup debug log level I see only thousands of errors like:

[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1837): proxy: worker http://1.2.3.4:10080/ already initialized
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1934): proxy: initialized single connection worker 12779 in child 2508 for (1.2.3.4)
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 12778 in child 2508 for worker http://1.2.3.4:10080/
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1837): proxy: worker http://1.2.3.4:10080/ already initialized
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1934): proxy: initialized single connection worker 12778 in child 2508 for (1.2.3.4)
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 12777 in child 2508 for worker http://1.2.3.4:10080/
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1837): proxy: worker http://1.2.3.4:10080/ already initialized
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1934): proxy: initialized single connection worker 12777 in child 2508 for (1.2.3.4)
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 12776 in child 2508 for worker http://1.2.3.4:10080/
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1837): proxy: worker http://1.2.3.4:10080/ already initialized
[Fri Apr 01 11:32:27 2011] [debug] proxy_util.c(1934): proxy: initialized single connection worker 12776 in child 2508 for (1.2.3.4)


When I make same steps with latest 2.0.64 it seems that it works fine (only seems because I don't forward traffic to this box).

Please let me know what all informatin should be provided.

OS is latest RedHat 5.6
Comment 1 Jacob 2014-04-08 13:38:13 UTC
We having similar memory issues at around 4k virtual hosts, on 2.2.26. Each child process consumes 3GB of memory, so on our idle server there is 9GB dedicated to Apache. Do we need to downgrade to circumvent this issue?
Comment 2 William A. Rowe Jr. 2018-11-07 21:09:37 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.