Bug 62907 - high virtual address space per httpd process
Summary: high virtual address space per httpd process
Status: NEEDINFO
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.37
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-13 10:34 UTC by Armin Abfalterer
Modified: 2018-11-13 11:25 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Abfalterer 2018-11-13 10:34:13 UTC
Hi all

we have observed a huge virtual address space (~20 GB) per httpd process on one of our productive systems (Red Hat 6.5, amd64).

Trying to reproduce the situation have not yet succeeded, nevertheless we can observe a quite high address space on a test system with Fedora 29 and standard httpd (installed from Red Hat repository).

[root@localhost ~]# cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)

[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.17-300.fc29.x86_64 #1 SMP Mon Nov 5 17:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# /usr/sbin/httpd -V
Server version: Apache/2.4.37 (Fedora)
Server built:   Nov  7 2018 11:56:38
Server's Module Magic Number: 20120211:83
Server loaded:  APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

The high address space seems to be related to the memory pool creation per thread of glbic. At least it can be shown that the space decreases when the number of threads are decreased. For example, with standard configuration from Red Hat following can be observed

[root@localhost ~]# ps -o comm,pid,ppid,vsz,rss -C httpd
COMMAND           PID  PPID    VSZ   RSS
httpd           29227     1  38768 13744
httpd           29228 29227  48940  7220
httpd           29229 29227 1106848 9324
httpd           29230 29227 1237980 9324
httpd           29231 29227 1106848 9324

[root@localhost ~]# cat /proc/29228/status | grep Threads
Threads:        1

[root@localhost ~]# cat /proc/29229/status | grep Threads
Threads:        65

Note the difference of VSZ between process 29228 (one thread) and 29229 (65 threads). Reducing the number of threads per process with following settings...

H2MaxWorkers 1
ThreadsPerChild 1
ThreadLimit 1
MaxRequestWorkers 1
ServerLimit 2

...leads to these results

[root@localhost ~]# ps -o comm,pid,ppid,vsz,rss -C httpd
COMMAND           PID  PPID    VSZ   RSS
httpd           30693     1  38432 13528
httpd           30694 30693  48940  7216
httpd           30695 30693 417804  8436

[root@localhost ~]# cat /proc/30695/status | grep Threads
Threads:        21

However, limiting the number of memory arenas by exporting MALLOC_ARENA_MAX (e.g. to 2) does not show any difference.

I know that virtual address space is not likely to run out on a 64-bit machine. Anyway I'd like to ask for your opinion how the address space can grow up to 20 GB?
Comment 1 Stefan Eissing 2018-11-13 11:25:37 UTC
Use "ulimit -s" to see what the default stack size per thread is on your system. On my ubuntu 64bit it is 8192. So every thread, regardless of what else is allocated for it, will have a 8MB stack.

With 50 threads per process and 3 worker children, that would sum up to 1.2 GB on virtual stack space alone.

See also: https://unix.stackexchange.com/questions/127602/default-stack-size-for-pthreads#280865

Does this match your observations?