Bug 64195

Summary: Tomcat NIO SSL Connector broken under Linux in Tomcat 9.0.31
Product: Tomcat 10 Reporter: Michael Seele <mseele>
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED DUPLICATE    
Severity: normal CC: krishnan212, sebastian.ratz, vinkvii
Priority: P2    
Version: 10.0.0-M1   
Target Milestone: ------   
Hardware: PC   
OS: Linux   
Attachments: SendBackServlet
DemoClient

Description Michael Seele 2020-03-04 12:37:35 UTC
Since Tomcat 9.0.31 we got multiple issues transfering files with org.apache.coyote.http11.Http11NioProtocol having SSL enabled.

The issues are completely gone when we switch to org.apache.coyote.http11.Http11Nio2Protocol or go back to Tomcat 9.0.30.

I've created a demo servlet that just read the incomming bytes and write it back to the output stream.
I've also implemented a demo client who sends requests to that servlet, reads the response and checks it.

Here you can find a tomcat docker image where everything is included:

- https://drive.google.com/open?id=1lLLt8y4hY2PCV-K81RBsT_2kPxYXOCv2 (used "docker save" command)
- https://drive.google.com/open?id=1vQ4Cwr7__Skx7YsJBk8oLo_-XXih_OLJ (used "docker export" command)

I've used the tag 9.0.31-jdk11-adoptopenjdk-hotspot from https://hub.docker.com/_/tomcat

You just need to run the client against that docker image.
I've added two SSL connectors:
Http11NioProtocol on port 8443 and Http11Nio2Protocol on port 8444

Running against Http11Nio2Protocol workes fine, running against Http11NioProtocol fails most of the time.

Here are a few of the different exceptions we got in our live applications using tomcat 9.0.31 & Http11NioProtocol:

Broken pipe (Write failed)
org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at de.guhsoft.guhsync.client.shared.ServerConnection.execute(ServerConnection.java:183)
        at de.guhsoft.guhsync.client.shared.calls.ServerCall.execute(ServerCall.java:30)
        at de.guhsoft.guhsync.client.shared.SyncService.run(SyncService.java:80)
        at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:108)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        ... 8 more
Caused by: javax.net.ssl.SSLException: Broken pipe (Write failed)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:987)
        at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
        at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
        at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
        at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
        at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:144)
        at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
        at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        ... 10 more
        Suppressed: java.net.SocketException: Broken pipe (Write failed)
                at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
                at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
                at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
                at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
                at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:351)
                ... 25 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
        at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
        at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:319)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:982)
        ... 22 more

org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
    java.lang.IllegalArgumentException: Request header is too large
        at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:722)
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:401)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834) 


StandardRequestService: An IncompatibleRemoteServiceException was thrown while processing this call.
    com.google.gwt.user.client.rpc.IncompatibleRemoteServiceException: Too few tokens in RPC request
        at com.google.gwt.user.server.rpc.RPC.decodeRequest(RPC.java:323)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:296)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
        at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)
    Caused by: com.google.gwt.user.client.rpc.SerializationException: Too few tokens in RPC request
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.extract(ServerSerializationStreamReader.java:997)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readBoolean(ServerSerializationStreamReader.java:516)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$1.readValue(ServerSerializationStreamReader.java:101)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader.readValue(ServerSerializationStreamReader.java:170)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:431)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:864)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeClass(ServerSerializationStreamReader.java:833)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:906)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54)
        at com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434)
        at com.google.gwt.user.server.rpc.RPC.decodeRequest(RPC.java:312)
        ... 27 more
    Caused by: java.lang.IndexOutOfBoundsException: Index 1930 out of bounds for length 1930
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:458)
        at com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.extract(ServerSerializationStreamReader.java:995)
        ... 151 more
Comment 1 Michael Seele 2020-03-04 12:38:10 UTC
Created attachment 37058 [details]
SendBackServlet
Comment 2 Michael Seele 2020-03-04 12:38:22 UTC
Created attachment 37059 [details]
DemoClient
Comment 3 Remy Maucherat 2020-03-04 12:50:03 UTC
*** Bug 64194 has been marked as a duplicate of this bug. ***
Comment 4 Remy Maucherat 2020-03-04 13:14:59 UTC
I have reverted the change in Tomcat 9.0.31 that probably caused this (the fix will be in 9.0.32), since I cannot immediately investigate. Given the plan for the Tomcat 9 branch, it is best to permanently defer this change to Tomcat 10 and 9.10.

I will work on integrating your testcase into the Tomcat testsuite and will see how well the problem can be reproduced.
The stacktraces provided are not useful, same as with 64195.
Comment 5 Michael Seele 2020-03-04 13:22:38 UTC
(In reply to Remy Maucherat from comment #4)
> I have reverted the change in Tomcat 9.0.31 that probably caused this (the
> fix will be in 9.0.32), since I cannot immediately investigate. Given the
> plan for the Tomcat 9 branch, it is best to permanently defer this change to
> Tomcat 10 and 9.10.
> 
> I will work on integrating your testcase into the Tomcat testsuite and will
> see how well the problem can be reproduced.
> The stacktraces provided are not useful, same as with 64195.

Hi Remy, thank you. Please see the referenced docker images (where a full tomcat installation takes place and is fully setup to reproduce this bug). Just need to run the docker image and execute the client against:
- import it via docker load (https://docs.docker.com/engine/reference/commandline/load/)
- run it via e.g. docker run --detach -it -p 8888:8443 -p 8889:8444 --rm --name tomcat <image_name>
- run DemoClient (e.g. java DemoClient https://localhost:8888/tomcat_nio
and you can reproduce the bug.

I can send you the docker images or an export of the tomcat also via email if you want?
Comment 6 Remy Maucherat 2020-03-04 18:10:40 UTC
This will stay in NEEDINFO. I added a simple test to the testsuite, which works for me. I will monitor the CI results for failures.
Comment 7 Mark Thomas 2020-03-04 18:36:35 UTC
I have a similar test case that also works.

I have also replicated the test servlet and client provided in Tomcat's unit tests and that works as expected as well.
Comment 8 Michael Seele 2020-03-09 07:40:03 UTC
(In reply to Mark Thomas from comment #7)
> I have a similar test case that also works.
> 
> I have also replicated the test servlet and client provided in Tomcat's unit
> tests and that works as expected as well.

Under which OS and JRE are you running the unit tests? Only happens for us under linux.
Is your unit test ssl enabled? Without ssl enabled, i don't get the error (like i wrote in the beginnging bug report).
We experience it in several applications which are live and operating.
And my test servlet & demo client in the docker image can reproduce the error always.

Do you need the server.xml configuration and the keys? Maybe it's an encryption problem?
Comment 9 Remy Maucherat 2020-03-09 09:39:51 UTC
None of that would help. If you want to help, you can try the 9.0.32 RC.
https://dist.apache.org/repos/dist/dev/tomcat/tomcat-9/v9.0.32/

I may change the trunk code to use a latch for now since it's easier to read.
Comment 10 Michael Seele 2020-03-09 11:07:14 UTC
(In reply to Remy Maucherat from comment #9)
> None of that would help. If you want to help, you can try the 9.0.32 RC.
> https://dist.apache.org/repos/dist/dev/tomcat/tomcat-9/v9.0.32/
> 
> I may change the trunk code to use a latch for now since it's easier to read.

Why does it not help giving you the same environment in which we reproduce the bug? What info (bug is on NEEDINFO) do you need then?

Anyway, i've tried 9.0.32 RC and can confirm that the bug is gone for 9.0.32 RC. I cannot reproduce it anymore.

But my attached demo is running way slower than under 9.0.30 (i sometimes kind of stops for a second or so).
I ran the demo 10 times and got the following durations with 9.0.32 RC:
21 sec / 8 sec / 4 sec / 5 sec / 12 sec / 6 sec / 5 sec / 10 sec / 10 sec / 8 sec
Average is 8,9 sec.

Running the demo 10 times under 9.0.30, each run takes 4 sec. 
Avagare is 4 sec.

FYI, i measured in nanos but i thought its clearer to use secs here
Comment 11 Michael Seele 2020-03-09 12:23:24 UTC
FYI: trying the demo against 10.0.0-M2 (https://dist.apache.org/repos/dist/dev/tomcat/tomcat-10/v10.0.0-M2/bin/) i always got this result:

[ERROR] RESULT NOT EQUAL: expected 1048576 / got: 736

Let me know if I can help you in any way...
Comment 12 Mark Thomas 2020-03-09 15:56:32 UTC
Marking as a duplicate of 64202 as it is sufficiently similar to this bug (issue is with NIO/SSL, issue appeared in 9.0.31, issue is fixed in 9.0.32) and the test case provided in 64202 allows the issue to be recreated with a local Tomcat.

*** This bug has been marked as a duplicate of bug 64202 ***
Comment 13 Mark Thomas 2020-03-16 17:19:27 UTC
*** Bug 64232 has been marked as a duplicate of this bug. ***
Comment 14 mgrigorov 2020-06-02 10:29:22 UTC
*** Bug 64486 has been marked as a duplicate of this bug. ***