Created attachment 36794 [details] Code to reproduce the problem (see description) Resource leak: under certain conditions, request objects related to WebSockets are not freed When Tomcat 8.5.38 is setting up a WebSocket (WS) connection with a client (or has just set up the connection - not sure), and then receives a TCP RST on that connection, it is possible that the associated objects are never freed. The objects are of the classes below. org.apache.tomcat.websocket.server.WsHandshakeRequest org.apache.catalina.connector.Request org.apache.coyote.Request org.apache.coyote.RequestInfo org.apache.catalina.connector.RequestFacade We saw this happen in production, and we were able to reproduce this with test code, running against our application, and against an out-of-the-box (OOB) (embedded) Tomcat. I have attached the stack traces of the two use cases (our application and OOB Tomcat). Interestingly, the stack traces are different. To reproduce the problem in a test environment, we have modified a TCP proxy to send a RST packet to the server shortly after sending the WebSocket upgrade HTTP request. When opening many WS connections, and having them automatically interrupted with RST packets, after a while a number of objects seem to be stuck in memory (see screenshot requests_objects.png). The objects stay in memory even when the proxy and client are shut down. Thank you for having a look at this. This failure mode does not happen often, but when it happens, it eventually can bring the JVM down because of memory pressure. ATTACHMENT The attachment contains: tomcat-webserver: an OOB (embedded) Tomcat with a WS endpoint websockets/tcp-proxy: a TCP proxy, modified to send RST packets - run ProxyMain to start the proxy websockets/websockets-client: a simple WS client, opening many connections - run SadPath to reproduce the problem requests_objects.png: a VisualVM screenshot showing stuck objects web_socket_connection_reset.txt: two stack traces (the first when reproducing the problem with our application, the second when reproducing the problem with tomcat-webserver)
Please re-test this with the latest 8.5.x release (8.5.46 as I type this) and confirm whether or not this resolves the issue. If it does not resolve the issue please update the test case and we will investigate.
Created attachment 36798 [details] Application code, using embedded Tomcat 8.5.46
We have retested using Tomcat 8.5.46 (new code attached), and could still reproduce the problem, following the steps below. Start the Tomcat application Start the proxy Start the test client (SadPath) Wait for 20 minutes Shut down the client Shut down the proxy Take a heapdump of the Tomcat application In the heapdump, observe 100+ instances of each of the classes below org.apache.coyote.Request org.apache.catalina.connector.Request org.apache.coyote.RequestInfo org.apache.tomcat.websocket.server.WsHandshakeRequest org.apache.catalina.connector.RequestFacade
Thanks for the report and the test case. It makes it so much easier to track down and fix bugs when you have this much information to start from. For the record, it was not what would normally a resource leak. More a failure to clean-up in a timely manner. Given enough (error free) further processing, the objects would have been cleaned up. Fixed in: - master for 9.0.27 onwards - 8.5.x for 8.5.47 onwards - 7.0.x for 7.0.97 onwards
Thank you for addressing this issue so swiftly. We are impressed. Keep up the good work!