Hi all

we have observed a huge virtual address space (~20 GB) per httpd process on one of our productive systems (Red Hat 6.5, amd64).

Trying to reproduce the situation have not yet succeeded, nevertheless we can observe a quite high address space on a test system with Fedora 29 and standard httpd (installed from Red Hat repository).

[root@localhost ~]# cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)

[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.17-300.fc29.x86_64 #1 SMP Mon Nov 5 17:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# /usr/sbin/httpd -V
Server version: Apache/2.4.37 (Fedora)
Server built:   Nov  7 2018 11:56:38
Server's Module Magic Number: 20120211:83
Server loaded:  APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

The high address space seems to be related to the memory pool creation per thread of glbic. At least it can be shown that the space decreases when the number of threads are decreased. For example, with standard configuration from Red Hat following can be observed

[root@localhost ~]# ps -o comm,pid,ppid,vsz,rss -C httpd
COMMAND           PID  PPID    VSZ   RSS
httpd           29227     1  38768 13744
httpd           29228 29227  48940  7220
httpd           29229 29227 1106848 9324
httpd           29230 29227 1237980 9324
httpd           29231 29227 1106848 9324

[root@localhost ~]# cat /proc/29228/status | grep Threads
Threads:        1

[root@localhost ~]# cat /proc/29229/status | grep Threads
Threads:        65

Note the difference of VSZ between process 29228 (one thread) and 29229 (65 threads). Reducing the number of threads per process with following settings...

H2MaxWorkers 1
ThreadsPerChild 1
ThreadLimit 1
MaxRequestWorkers 1
ServerLimit 2

...leads to these results

[root@localhost ~]# ps -o comm,pid,ppid,vsz,rss -C httpd
COMMAND           PID  PPID    VSZ   RSS
httpd           30693     1  38432 13528
httpd           30694 30693  48940  7216
httpd           30695 30693 417804  8436

[root@localhost ~]# cat /proc/30695/status | grep Threads
Threads:        21

However, limiting the number of memory arenas by exporting MALLOC_ARENA_MAX (e.g. to 2) does not show any difference.

I know that virtual address space is not likely to run out on a 64-bit machine. Anyway I'd like to ask for your opinion how the address space can grow up to 20 GB?