Bug 60551

Summary: Spawning more workers until it cannot fork a new process
Product: Apache httpd-2 Reporter: Ian Eggleton <ian.eggleton>
Component: mpm_workerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: NEW ---    
Severity: critical CC: szg0000
Priority: P2    
Version: 2.4.25   
Target Milestone: ---   
Hardware: Other   
OS: AIX   

Description Ian Eggleton 2017-01-04 17:01:22 UTC
Upgrading from apache 2.4.20 to 23 or 25.
Even when the services are idle, only receiving a healthcheck request every 2 seconds from a loadbalancer, the server reports it is initializing  a new worker every half hour to 45 minutes for about 2 days and the number of apache processes slowly creeps up until it hits one of the ulimits

Error Log output
AH00925: initializing worker https://<server>/ shared
[Sat Dec 24 21:05:58.371791 2016] [proxy:debug] [pid 33423390:tid 1] proxy_util.c(1821): AH00927: initializing worker https://t<server>/ local
[Sat Dec 24 21:05:58.371859 2016] [proxy:debug] [pid 33423390:tid 1] proxy_util.c(1856): AH00930: initialized pool in child 33423390 for <server>
[mpm_worker:error] [pid 19398908:tid 1] (11)Resource temporarily unavailable:
AH00283: fork: Unable to fork new process

OS AIX 6.1SP9
Compile parameters
export LIBPATH="/opt/freeware/lib:/usr/lib:/usr/openssl_101s_64/lib:/usr/local/lib:/usr/local/pcre_64/lib"
export LDFLAGS="-L/opt/freeware/lib -L/usr/openssl_101s_64/lib -L/usr/local/pcre_64/lib -Wl,-bbigtoc"
export CC="gcc"
export CFLAGS="-maix64 -O2 -pthread"
export OBJECT_MODE=64


./configure --prefix=%{prefix} \
        --enable-so \
        --enable-static-support \
        --enable-deflate \
        --enable-mods-static='few ratelimit include sed mime log_debug headers setenvif watchdog proxy proxy_connect proxy_http proxy_ftp proxy_ajp proxy_balancer session session_cookie vhost_alias alias rewrite slotmem_shm socache_dbm' \
        --with-included-apr \
        --with-included-apr-util \
        --with-z=/usr/local/zlib_64 \
        --enable-nonportable-atomics=yes \
        --with-libxml2=/usr/local/lib \
        --enable-ssl \
        --enable-ssl-staticlib-deps \
        --with-ssl=/usr/openssl_101s_64 \
        --with-pcre=/usr/local/pcre_64
Comment 1 Eric Covener 2017-01-04 18:03:14 UTC
what mpm config do you have?
Comment 2 Ian Eggleton 2017-01-05 10:28:29 UTC
(In reply to Eric Covener from comment #1)
> what mpm config do you have?

Good point, should have included that, although it's not changed from 2.20
Timeout 300
KeepAlive On
MaxKeepAliveRequests 600
KeepAliveTimeout 5
UseCanonicalName Off
HostnameLookups Off

ThreadsPerChild 25
StartServers 10
MinSpareThreads 250
MaxSpareThreads 300
ServerLimit 25
MaxClients 50
ListenBacklog 100
MaxRequestsPerChild 1000


and the compile details are

Server version: Apache/2.4.25 (Unix)
Server built:   Dec 22 2016 10:37:12
Server's Module Magic Number: 20120211:67
Server loaded:  APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture:   64-bit
Server MPM:     worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_core.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_core.c
  mod_access_compat.c
  mod_auth_basic.c
  mod_socache_dbm.c
  mod_so.c
  mod_watchdog.c
  mod_ratelimit.c
  mod_reqtimeout.c
  mod_include.c
  mod_filter.c
  mod_sed.c
  mod_deflate.c
  http_core.c
  mod_mime.c
  mod_log_config.c
  mod_log_debug.c
  mod_env.c
  mod_headers.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_fcgi.c
  mod_proxy_scgi.c
  mod_proxy_fdpass.c
  mod_proxy_wstunnel.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  mod_proxy_express.c
  mod_proxy_hcheck.c
  mod_session.c
  mod_session_cookie.c
  mod_session_dbd.c
  mod_slotmem_shm.c
  mod_ssl.c
  mod_lbmethod_byrequests.c
  mod_lbmethod_bytraffic.c
  mod_lbmethod_bybusyness.c
  mod_lbmethod_heartbeat.c
  worker.c
  mod_unixd.c
  mod_status.c
  mod_autoindex.c
  mod_vhost_alias.c
  mod_dir.c
  mod_alias.c
  mod_rewrite.c