Bug 60551 - Spawning more workers until it cannot fork a new process
Summary: Spawning more workers until it cannot fork a new process
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_worker (show other bugs)
Version: 2.4.25
Hardware: Other AIX
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-01-04 17:01 UTC by Ian Eggleton
Modified: 2017-03-02 11:24 UTC (History)
1 user (show)


Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ian Eggleton 2017-01-04 17:01:22 UTC 
Upgrading from apache 2.4.20 to 23 or 25.
Even when the services are idle, only receiving a healthcheck request every 2 seconds from a loadbalancer, the server reports it is initializing  a new worker every half hour to 45 minutes for about 2 days and the number of apache processes slowly creeps up until it hits one of the ulimits

Error Log output
AH00925: initializing worker https://<server>/ shared
[Sat Dec 24 21:05:58.371791 2016] [proxy:debug] [pid 33423390:tid 1] proxy_util.c(1821): AH00927: initializing worker https://t<server>/ local
[Sat Dec 24 21:05:58.371859 2016] [proxy:debug] [pid 33423390:tid 1] proxy_util.c(1856): AH00930: initialized pool in child 33423390 for <server>
[mpm_worker:error] [pid 19398908:tid 1] (11)Resource temporarily unavailable:
AH00283: fork: Unable to fork new process

OS AIX 6.1SP9
Compile parameters
export LIBPATH="/opt/freeware/lib:/usr/lib:/usr/openssl_101s_64/lib:/usr/local/lib:/usr/local/pcre_64/lib"
export LDFLAGS="-L/opt/freeware/lib -L/usr/openssl_101s_64/lib -L/usr/local/pcre_64/lib -Wl,-bbigtoc"
export CC="gcc"
export CFLAGS="-maix64 -O2 -pthread"
export OBJECT_MODE=64


./configure --prefix=%{prefix} \
        --enable-so \
        --enable-static-support \
        --enable-deflate \
        --enable-mods-static='few ratelimit include sed mime log_debug headers setenvif watchdog proxy proxy_connect proxy_http proxy_ftp proxy_ajp proxy_balancer session session_cookie vhost_alias alias rewrite slotmem_shm socache_dbm' \
        --with-included-apr \
        --with-included-apr-util \
        --with-z=/usr/local/zlib_64 \
        --enable-nonportable-atomics=yes \
        --with-libxml2=/usr/local/lib \
        --enable-ssl \
        --enable-ssl-staticlib-deps \
        --with-ssl=/usr/openssl_101s_64 \
        --with-pcre=/usr/local/pcre_64
Comment 1 Eric Covener 2017-01-04 18:03:14 UTC 
what mpm config do you have?
Comment 2 Ian Eggleton 2017-01-05 10:28:29 UTC 
(In reply to Eric Covener from comment #1)
> what mpm config do you have?

Good point, should have included that, although it's not changed from 2.20
Timeout 300
KeepAlive On
MaxKeepAliveRequests 600
KeepAliveTimeout 5
UseCanonicalName Off
HostnameLookups Off

ThreadsPerChild 25
StartServers 10
MinSpareThreads 250
MaxSpareThreads 300
ServerLimit 25
MaxClients 50
ListenBacklog 100
MaxRequestsPerChild 1000


and the compile details are

Server version: Apache/2.4.25 (Unix)
Server built:   Dec 22 2016 10:37:12
Server's Module Magic Number: 20120211:67
Server loaded:  APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture:   64-bit
Server MPM:     worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_core.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_core.c
  mod_access_compat.c
  mod_auth_basic.c
  mod_socache_dbm.c
  mod_so.c
  mod_watchdog.c
  mod_ratelimit.c
  mod_reqtimeout.c
  mod_include.c
  mod_filter.c
  mod_sed.c
  mod_deflate.c
  http_core.c
  mod_mime.c
  mod_log_config.c
  mod_log_debug.c
  mod_env.c
  mod_headers.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_fcgi.c
  mod_proxy_scgi.c
  mod_proxy_fdpass.c
  mod_proxy_wstunnel.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  mod_proxy_express.c
  mod_proxy_hcheck.c
  mod_session.c
  mod_session_cookie.c
  mod_session_dbd.c
  mod_slotmem_shm.c
  mod_ssl.c
  mod_lbmethod_byrequests.c
  mod_lbmethod_bytraffic.c
  mod_lbmethod_bybusyness.c
  mod_lbmethod_heartbeat.c
  worker.c
  mod_unixd.c
  mod_status.c
  mod_autoindex.c
  mod_vhost_alias.c
  mod_dir.c
  mod_alias.c
  mod_rewrite.c