Bug 69123 - memory leak in worker proxy pool
Summary: memory leak in worker proxy pool
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.4.59
Hardware: PC Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-06-07 09:17 UTC by Armin Abfalterer
Modified: 2024-06-12 09:27 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Armin Abfalterer 2024-06-07 09:17:25 UTC
A memory leak has been identified in httpd version 2.4.59 running in prefork mode. Single child processes were observed holding hundreds of MB of heap memory. 

Memory/pool debugging with gdb revealed that a large number of proxy_worker_cp pools are being allocated with no free blocks remaining in the allocator. This suggests that proxy_worker_cp pools (or sub-pools) are not being properly released.

VmFlags: rd wr mr mw me ac 
095da000-3151d000 rw-p 00000000 00:00 0                                  [heap]
Size:             654604 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:              639356 kB
Pss:              639356 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:    639356 kB
Referenced:       638492 kB
Anonymous:        639356 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                532 kB
SwapPss:             532 kB
Locked:                0 kB
THPeligible:    0
ProtectionKey:         0

> dump_all_pools
 Pool 'apr_global_pool' [0x89ebd68]: 7288/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
     Pool 'No tag' [0x8aafa68]: 8056/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
     Pool 'No tag' [0x94c1868]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
     Pool 'No tag' [0x94ae708]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
     Pool 'No tag' [0x946e6e8]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
     Pool 'No tag' [0x946a528]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
	 
	 < ... truncated ... >
	 
     Pool 'process' [0x89efd88]: 22160/172032 free (21 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
         Pool 'stderr_pool' [0x935bbf8]: 7416/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
         Pool 'plog' [0x8a3d6c8]: 4512/212992 free (26 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
         Pool 'pcommands' [0x8a15f28]: 7936/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
         Pool 'pconf' [0x89f1d98]: 76616/8396800 free (1025 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
             Pool 'pchild' [0x95c6f68]: 4216/548864 free (67 blocks) allocator: 0x95989a0 free blocks in allocator: 32764 kiB
                 Pool 'rewrite_cachep' [0x96a9678]: 7992/8192 free (1 blocks) allocator: 0x95989a0 free blocks in allocator: 32764 kiB
                 Pool 'proxy_worker_cp' [0x96a3648]: 8104/8192 free (1 blocks) allocator: 0x8f04fb0 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x96a5658]: 8032/8192 free (1 blocks) allocator: 0x8f04fb0 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x96a7668]: 8104/8192 free (1 blocks) allocator: 0x8f04fb0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x96a1638]: 8104/8192 free (1 blocks) allocator: 0x8f04f40 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x96191f8]: 8104/8192 free (1 blocks) allocator: 0x8f04ed0 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x961b208]: 8032/8192 free (1 blocks) allocator: 0x8f04ed0 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x961d218]: 8104/8192 free (1 blocks) allocator: 0x8f04ed0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x96171e8]: 8104/8192 free (1 blocks) allocator: 0x959ac50 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x96111b8]: 8104/8192 free (1 blocks) allocator: 0x959abe0 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x96131c8]: 8032/8192 free (1 blocks) allocator: 0x959abe0 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x96151d8]: 8104/8192 free (1 blocks) allocator: 0x959abe0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x960f1a8]: 8104/8192 free (1 blocks) allocator: 0x959ab70 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x9609178]: 8104/8192 free (1 blocks) allocator: 0x959ab00 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x960b188]: 8016/8192 free (1 blocks) allocator: 0x959ab00 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x960d198]: 8104/8192 free (1 blocks) allocator: 0x959ab00 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x9607168]: 8104/8192 free (1 blocks) allocator: 0x959aa90 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x9740f88]: 7880/8192 free (1 blocks) allocator: 0x959aa90 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x9601138]: 8104/8192 free (1 blocks) allocator: 0x959aa20 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x9603148]: 8032/8192 free (1 blocks) allocator: 0x959aa20 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x9605158]: 8104/8192 free (1 blocks) allocator: 0x959aa20 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95ff128]: 8104/8192 free (1 blocks) allocator: 0x959a9b0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x95f90f8]: 8104/8192 free (1 blocks) allocator: 0x959a940 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x95fb108]: 8032/8192 free (1 blocks) allocator: 0x959a940 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x95fd118]: 8104/8192 free (1 blocks) allocator: 0x959a940 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95f70e8]: 8104/8192 free (1 blocks) allocator: 0x959a8d0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x95f10b8]: 8104/8192 free (1 blocks) allocator: 0x959a860 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x95f30c8]: 8032/8192 free (1 blocks) allocator: 0x959a860 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x95f50d8]: 8104/8192 free (1 blocks) allocator: 0x959a860 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95ef0a8]: 8104/8192 free (1 blocks) allocator: 0x959a7f0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x95e9078]: 8104/8192 free (1 blocks) allocator: 0x959a780 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x95eb088]: 8016/8192 free (1 blocks) allocator: 0x959a780 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x95ed098]: 8104/8192 free (1 blocks) allocator: 0x959a780 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95e7068]: 8104/8192 free (1 blocks) allocator: 0x959a710 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x974cfe8]: 7880/8192 free (1 blocks) allocator: 0x959a710 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x95e1038]: 8104/8192 free (1 blocks) allocator: 0x959a450 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x95e3048]: 8016/8192 free (1 blocks) allocator: 0x959a450 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x95e5058]: 8104/8192 free (1 blocks) allocator: 0x959a450 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95df028]: 8104/8192 free (1 blocks) allocator: 0x959a3e0 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x984bce8]: 7880/8192 free (1 blocks) allocator: 0x959a3e0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x95d8ff8]: 8104/8192 free (1 blocks) allocator: 0x959a370 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x95db008]: 8016/8192 free (1 blocks) allocator: 0x959a370 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x95dd018]: 8104/8192 free (1 blocks) allocator: 0x959a370 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95d6fe8]: 8104/8192 free (1 blocks) allocator: 0x959a300 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x97d8258]: 7880/8192 free (1 blocks) allocator: 0x959a300 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x95d0fb8]: 8104/8192 free (1 blocks) allocator: 0x9599110 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x95d2fc8]: 8032/8192 free (1 blocks) allocator: 0x9599110 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x95d4fd8]: 8104/8192 free (1 blocks) allocator: 0x9599110 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x95cefa8]: 8104/8192 free (1 blocks) allocator: 0x9599440 free blocks in allocator: 0 kiB
                 Pool 'transaction' [0x95caf88]: 8104/8192 free (1 blocks) allocator: 0x95989a0 free blocks in allocator: 32764 kiB
             Pool 'wd_server_conf' [0x95bd238]: 7976/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
                 Pool 'No tag' [0x8aa3a08]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
             Pool 'No tag' [0x944e0e8]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
             Pool 'proxy_server_conf' [0x92f8f28]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
             Pool 'proxy_server_conf' [0x9019898]: 8104/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
             Pool 'No tag' [0x8a43fb8]: 8032/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
             Pool 'proxy_server_conf' [0x8a33338]: 2224/8192 free (1 blocks) allocator: 0x89ebce0 free blocks in allocator: 632 kiB
                 Pool 'proxy_worker_cp' [0x98404a8]: 8104/8192 free (1 blocks) allocator: 0x8e49380 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x98458b8]: 8016/8192 free (1 blocks) allocator: 0x8e49380 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x98478c8]: 8104/8192 free (1 blocks) allocator: 0x8e49380 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x983e498]: 8104/8192 free (1 blocks) allocator: 0x9724950 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x98498d8]: 7848/8192 free (1 blocks) allocator: 0x9724950 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x9836458]: 8104/8192 free (1 blocks) allocator: 0x9503750 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x9838468]: 8016/8192 free (1 blocks) allocator: 0x9503750 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x983a478]: 8104/8192 free (1 blocks) allocator: 0x9503750 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x9834448]: 8104/8192 free (1 blocks) allocator: 0x97251f0 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x983c488]: 7848/8192 free (1 blocks) allocator: 0x97251f0 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x982a548]: 8104/8192 free (1 blocks) allocator: 0x947b580 free blocks in allocator: 0 kiB
                     Pool 'proxy_conn_pool' [0x982e418]: 8016/8192 free (1 blocks) allocator: 0x947b580 free blocks in allocator: 0 kiB
                         Pool 'proxy_conn_scpool' [0x9830428]: 8104/8192 free (1 blocks) allocator: 0x947b580 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_dns' [0x9828538]: 8104/8192 free (1 blocks) allocator: 0x9591f00 free blocks in allocator: 0 kiB
                     Pool 'No tag' [0x9832438]: 7832/8192 free (1 blocks) allocator: 0x9591f00 free blocks in allocator: 0 kiB
                 Pool 'proxy_worker_cp' [0x98204f8]: 8104/8192 free (1 blocks) allocator: 0x947e0d0 free blocks in allocator: 0 kiB
				 
				 < ... truncated ... >
				
Total 2716704/12009472 free (1466 blocks)


> dump_one_pool ((apr_pool_t*))0x95c6f68
Pool 'pchild' [0x95c6f68]: 4216/548864 free (67 blocks)

The memory leak appears to be isolated to a specific virtual host configuration. Other virtual hosts do not exhibit this behavior. This suggests a potential interaction between httpd and the backend application configured for the specific virtual host.
Comment 1 Armin Abfalterer 2024-06-12 09:27:19 UTC
friendly waving over with the request that someone take a quick look at the problem described