Created attachment 35564 [details]
jstack thread dump

We are observing a scenario when NIO2 connector on tomcat becomes unresponsive after some period of time, at the same time NIO connector running on the same host is still able to process the same requests and serves traffic. Only server restart helps in this case.
This issue is intermittent and with the current infrastructure we have few nodes behind LB and it happens from time to time (like once per week) for each node, so it seems to be not a node or hardware specific in our case.
Below is our server.xml:

    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" maxThreads="800" minSpareThreads="100"/>


      <Connector executor="tomcatServiceThreadPool"
                 port="8080"
                 protocol="org.apache.coyote.http11.Http11Nio2Protocol"
                 connectionTimeout="1000"
                 enableLookups="false"
                 acceptorThreadCount="1"
                 processorCache="800"
                 socket.tcpNoDelay="true"
                 socket.soKeepAlive="true"
                 socket.soLingerOn="false"
                 compression="256"
                 compressableMimeType="text/html,text/xml,text/plain,application/x-protobuf,application/json,application/javascript"
                 URIEncoding="UTF-8" />

      <!-- The load balancer terminates SSL connections and
           then forwards them to the following connector as
           normal HTTP (non-secure) requests
       -->
      <Connector executor="tomcatServiceThreadPool"
                 port="8443"
                 protocol="org.apache.coyote.http11.Http11NioProtocol"
                 connectionTimeout="1000"
                 enableLookups="false"
                 connectionLinger="-1"
                 acceptorThreadCount="20"
                 processorCache="800"
                 socket.tcpNoDelay="true"
                 socket.soKeepAlive="true"
                 socket.soLingerOn="false"
                 compression="256"
                 compressableMimeType="text/html,text/xml,text/plain,application/x-protobuf,application/json,application/javascript"
                 URIEncoding="UTF-8" />


      <!-- Define an AJP 1.3 Connector on port 8009 -->
      <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

Also below is an example of the behavior we observe:

curl -verbose 'http://localhost:8080/rs?id=nio2issue'
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /rs?id=nio2issue HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8080
> Accept: */*
> Referer: rbose
>
* Closing connection #0
* Failure when receiving data from the peer
curl: (56) Failure when receiving data from the peer

at the same time:

curl -i 'http://localhost:8443/rs?id=nio2issue'
HTTP/1.1 302 Found

Also, no unusual errors are logged to catalina.out at the time of the accident. Enclosed is thread dump from the server.
Also, we have observed the same behavior on tomcat 8.0.18 and upgraded to the latest version in the same release 8.0.47 but it didn't help.

Please let me know what else might be helpful as we keep one of the servers in this state, for now, to be able to gather any data as the issue is intermittent and we were not able to reproduce with a simple load test.

Regards,
Oleg.