Bug 61918

CentOS 6.6
Tomcat 8.5.16
Connector protocol: org.apache.coyote.http11.Http11Nio2Protocol

We have noticed that the connectionCount jmx value for the nio2 connector trends upwards on our servers during the week and never drops down to 1. 

When we used netstat to trace these connections, we found the expected handful of connections active but not the value shown by jmx. We also enabled tomcat debug logging to see if any exceptions occurred during socket operations but found nothing of interest.

We believe that our data center's network configuration is causing sockets to be closed independently of the Tomcat Coyote handling but do not know what specifically is triggering it. We also found a suspicious line in Nio2Endpoint connector which could explain this false floor that keeps rising. 

I can artificially re-create this situation by forcefully closing the socket in a Spring MVC controller handling. Every time this endpoint is called, we prevent Tomcat from decrementing the connection count upon close. See Nio2Endpoint.closeSocket, line 358, only decrements the connection limit latch when the socket is open. 

Please see the attachment for:
* MyController.java - artificial recreation using Spring mvc and test framework (easily placed in a servlet if you also include org.springframework:spring-test module from maven)
* jmx.png - screenshot of jmx statistic we are monitoring
* connectionCounts.png - dashboard view of our server pool connection count for a week
* Nio2Endpoint.java.patch - proposed patch for this situation