Starting with Tomcat 9.0.29 (but not 9.0.27, and fixed by reverting), the Java process builds-up open files until hitting the Linux default limit of 4096. lsof shows that the files are all those that would be checked when determining if a JSP file needs to be recompiled (the JSP file, its includes, and all related *.tld files). With "development=true" (the default), the files build-up in proportion to the request load. With "development=false", the files build-up more consistently (around 100 files every 10 seconds in our case). When a garbage collection occurs, the open file count drops back to around 100. With our heap-space of 1024 MB, combined with some include-heavy JSP pages, we run out of file handles before garbage collection, and requests fail with errors. We increased the limits in /etc/security/limits.conf as a workaround, but the fix was to downgrade to Tomcat 9.0.27.
Thanks for the report. This is another regression in the fix for the race condition in JSP compilation that meant Jasper would see a modified JSP but then process stale content for that file from the static resource cache. I think I can address this with my favourite kind of fix (deleting some code) but I want to run a few more tests before I commit the fix.
Fixed in: - master for 9.0.30 onwards - 8.5.x for 8.5.50 onwards
I have confirmed this issue is resolved during our updates to Tomcat 9.0.30. Thank you for your help.
*** Bug 64857 has been marked as a duplicate of this bug. ***