Bug 61918 - Nio2Endpoint may miscount connectionLimitLatch while closing socket that is already closed
Summary: Nio2Endpoint may miscount connectionLimitLatch while closing socket that is a...
Alias: None
Product: Tomcat 8
Classification: Unclassified
Component: Connectors (show other bugs)
Version: 8.5.16
Hardware: PC All
: P2 normal (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2017-12-19 17:44 UTC by Ryan Fong
Modified: 2017-12-19 20:04 UTC (History)
0 users

data and patch (179.67 KB, application/x-zip-compressed)
2017-12-19 17:45 UTC, Ryan Fong

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Fong 2017-12-19 17:44:34 UTC
CentOS 6.6
Tomcat 8.5.16
Connector protocol: org.apache.coyote.http11.Http11Nio2Protocol

We have noticed that the connectionCount jmx value for the nio2 connector trends upwards on our servers during the week and never drops down to 1. 

When we used netstat to trace these connections, we found the expected handful of connections active but not the value shown by jmx. We also enabled tomcat debug logging to see if any exceptions occurred during socket operations but found nothing of interest.

We believe that our data center's network configuration is causing sockets to be closed independently of the Tomcat Coyote handling but do not know what specifically is triggering it. We also found a suspicious line in Nio2Endpoint connector which could explain this false floor that keeps rising. 

I can artificially re-create this situation by forcefully closing the socket in a Spring MVC controller handling. Every time this endpoint is called, we prevent Tomcat from decrementing the connection count upon close. See Nio2Endpoint.closeSocket, line 358, only decrements the connection limit latch when the socket is open. 

Please see the attachment for:
* MyController.java - artificial recreation using Spring mvc and test framework (easily placed in a servlet if you also include org.springframework:spring-test module from maven)
* jmx.png - screenshot of jmx statistic we are monitoring
* connectionCounts.png - dashboard view of our server pool connection count for a week
* Nio2Endpoint.java.patch - proposed patch for this situation
Comment 1 Ryan Fong 2017-12-19 17:45:15 UTC
Created attachment 35619 [details]
data and patch
Comment 2 Remy Maucherat 2017-12-19 20:04:52 UTC
I'm not convinced closing sockets like this is a really good idea. The patch may reintroduce more serious problems, but adding a closed flag seems possible.
This will be in 9.0.3 and 8.5.25.