While smoke testing NIO2 connector with following config: <Connector port="84" protocol="org.apache.coyote.http11.Http11Nio2Protocol" connectionTimeout="20000" redirectPort="8443" /> I get a number of warnings in my log files: 05-Jul-2015 22:48:07.685 WARNING [http-nio2-449-exec-3] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket. I don't find anything suspicious in my tests or configuration.
It happens both with HTTP (TCP port 84 in my tests) and HTTPS (TCP port 449): 05-Jul-2015 22:46:45.671 WARNING [http-nio2-449-exec-4] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket. 05-Jul-2015 22:46:46.667 WARNING [http-nio2-84-exec-3] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket. -Ognjen
This would happen occasionally for me. Now this doesn't but it still does rarely in the CI runs. I cannot figure out why it still fails after my fixes. Since it doesn't look like a very useful feature for NIO2 which should scale efficiently, I disabled maxConnections in that endpoint in trunk (forced to -1). Feel free to look into it if you think a fix would be useful.
Can I help somehow to get to the root cause of the problem? For instance, to print stack trace and connection count every time AbstractEndpoint.countUpOrAwaitConnection and .countDownConnection is executed? It seems to me that the problem easily reproducible on my environment.
Feel free to debug it, but as far I am concerned I am probably done with this feature at the moment.
I narrowed the problem to one single problematic example: /examples/servlets/nonblocking/numberwriter. It is example to demonstrate Servlet 3.1 Non-blocking IO. Crawling that single URL from Java using java.net.URL / URLConnection classes causes reported log messages.
(In reply to Ognjen Blagojevic from comment #5) > Crawling that single URL from Java using > java.net.URL / URLConnection classes causes reported log messages. s/Crawling/Reading/g
Here is a minimal test case: public static void main(String[] argv) throws Exception { URL url = new URL("http://localhost:84/examples/servlets/nonblocking/numberwriter"); InputStream urlStream = url.openStream(); byte b[] = new byte[1000]; int numRead = urlStream.read(b); while (numRead != -1) { numRead = urlStream.read(b); } urlStream.close(); } Log messages start to appear some 20-30s, after I execute this test.
I'll take a look at this. I think we do need to restore maxConnections support for NIO2 (probably with the limit disabled by default). While I am confident that NIO2 can scale to as many connections as the server can handle, applications may not be able to and in this case the ability to limit connections is useful. Thanks for the work to narrow this down to a simple test case.
Quick question. Do you see this on trunk or just 8.0.x?
I've just answered my own question. Running this test case once on Windows triggers the warning message once a second (roughly). Given how easy it is to reproduce, a fix should be (fairly) easy to track down.
Fixed in trunk, 8.0.x (for 8.0.25 onwards) and 7.0.x (for 7.0.64 onwards) for all connectors although the issue is only likely to be visible with NIO2 HTTP in 8.0.x. I also added the general protection to NIO2 in 8.0.x that was masking this issue in trunk.
*** Bug 58104 has been marked as a duplicate of this bug. ***
*** Bug 58105 has been marked as a duplicate of this bug. ***
Thanks for the fix, and +1 for restoring maxConnections now that it works properly. I was way too certain this was a NIO2 issue only I guess.