When doing a propfind on directories with many files (e.g. 100000), mod_dav consumes multiple GB of memory. The cause is that for each file, the prop_db gets opened and closed, but closing the propdb does not free any memory. There is a source code comment: /* Currently, mod_dav's pool usage doesn't allow clearing this pool. */ #if 0 apr_pool_destroy(propdb->p); #endif Even with the pool_destroy in place, the dav_propdb structure itself would not get destroyed, as it is not allocated from the pool. The original commit that added this pool was talking about 'lifetime issues'. Are they resolved yet? Or, on the other hand, would it be possible to open the propdb only once for the propfind request?
IMHO the lifetime issues are not fixed and fixing them is likely to require a severe rebuild of the pool handling in mod_dav including API changes for the providers. As PROPFIND possibly affects more than one resource and each resource has its own (possibly distinct) propdb it needs to be opened separately for each resource. Possibly fixable by a different design as well (one propdb for all resources).
Just for info I posted a series of patches to httpd 2.2.21 in bug #48130 that fix these memory consumption problems with mod_dav_fs when a lot of files are put into a single directory. They work well in our product.
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.