Bug 58103 - NIO2 warning: Incorrect connection count, multiple socket.close called on the same socket.
NIO2 warning: Incorrect connection count, multiple socket.close called on the...
Status: CLOSED FIXED
Product: Tomcat 8
Classification: Unclassified
Component: Connectors
8.0.x-trunk
PC All
: P2 minor (vote)
: ----
Assigned To: Tomcat Developers Mailing List
:
: 58104 58105 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2015-07-05 20:59 UTC by Ognjen Blagojevic
Modified: 2015-08-03 12:42 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ognjen Blagojevic 2015-07-05 20:59:19 UTC
While smoke testing NIO2 connector with following config:
             
    <Connector port="84" protocol="org.apache.coyote.http11.Http11Nio2Protocol"
               connectionTimeout="20000"
               redirectPort="8443" />

I get a number of warnings in my log files:

05-Jul-2015 22:48:07.685 WARNING [http-nio2-449-exec-3] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket.

I don't find anything suspicious in my tests or configuration.
Comment 1 Ognjen Blagojevic 2015-07-05 21:07:49 UTC
It happens both with HTTP (TCP port 84 in my tests) and HTTPS (TCP port 449):

05-Jul-2015 22:46:45.671 WARNING [http-nio2-449-exec-4] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket.
05-Jul-2015 22:46:46.667 WARNING [http-nio2-84-exec-3] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket.

-Ognjen
Comment 2 Remy Maucherat 2015-07-21 14:18:37 UTC
This would happen occasionally for me. Now this doesn't but it still does rarely in the CI runs.

I cannot figure out why it still fails after my fixes. Since it doesn't look like a very useful feature for NIO2 which should scale efficiently, I disabled maxConnections in that endpoint in trunk (forced to -1). Feel free to look into it if you think a fix would be useful.
Comment 3 Ognjen Blagojevic 2015-07-21 14:56:43 UTC
Can I help somehow to get to the root cause of the problem?

For instance, to print stack trace and connection count every time AbstractEndpoint.countUpOrAwaitConnection and .countDownConnection is executed?

It seems to me that the problem easily reproducible on my environment.
Comment 4 Remy Maucherat 2015-07-21 15:22:33 UTC
Feel free to debug it, but as far I am concerned I am probably done with this feature at the moment.
Comment 5 Ognjen Blagojevic 2015-07-23 00:05:29 UTC
I narrowed the problem to one single problematic example: /examples/servlets/nonblocking/numberwriter. It is example to demonstrate Servlet 3.1 Non-blocking IO. Crawling that single URL from Java using java.net.URL / URLConnection classes causes reported log messages.
Comment 6 Ognjen Blagojevic 2015-07-23 00:09:43 UTC
(In reply to Ognjen Blagojevic from comment #5)
> Crawling that single URL from Java using
> java.net.URL / URLConnection classes causes reported log messages.

s/Crawling/Reading/g
Comment 7 Ognjen Blagojevic 2015-07-23 07:20:26 UTC
Here is a minimal test case:

    public static void main(String[] argv) throws Exception {
        URL url = new URL("http://localhost:84/examples/servlets/nonblocking/numberwriter");
        InputStream urlStream = url.openStream();
        byte b[] = new byte[1000];
        int numRead = urlStream.read(b);
        while (numRead != -1) {
            numRead = urlStream.read(b);
        }
        urlStream.close();
    }

Log messages start to appear some 20-30s, after I execute this test.
Comment 8 Mark Thomas 2015-07-27 15:35:40 UTC
I'll take a look at this.

I think we do need to restore maxConnections support for NIO2 (probably with the limit disabled by default). While I am confident that NIO2 can scale to as many connections as the server can handle, applications may not be able to and in this case the ability to limit connections is useful.

Thanks for the work to narrow this down to a simple test case.
Comment 9 Mark Thomas 2015-07-27 18:26:13 UTC
Quick question. Do you see this on trunk or just 8.0.x?
Comment 10 Mark Thomas 2015-07-27 18:30:58 UTC
I've just answered my own question. Running this test case once on Windows triggers the warning message once a second (roughly). Given how easy it is to reproduce, a fix should be (fairly) easy to track down.
Comment 11 Mark Thomas 2015-07-28 13:49:43 UTC
Fixed in trunk, 8.0.x (for 8.0.25 onwards) and 7.0.x (for 7.0.64 onwards) for all connectors although the issue is only likely to be visible with NIO2 HTTP in 8.0.x.

I also added the general protection to NIO2 in 8.0.x that was masking this issue in trunk.
Comment 12 Mark Thomas 2015-07-28 13:55:34 UTC
*** Bug 58104 has been marked as a duplicate of this bug. ***
Comment 13 Mark Thomas 2015-07-28 13:57:01 UTC
*** Bug 58105 has been marked as a duplicate of this bug. ***
Comment 14 Remy Maucherat 2015-07-29 14:01:12 UTC
Thanks for the fix, and +1 for restoring maxConnections now that it works properly.

I was way too certain this was a NIO2 issue only I guess.