Sub-Component - Coyote OS : Redhat Linux Overview: Tomcat implemented to transport http/2 packets between 2 systems (h2c connection). Steps to Reproduce: With JMeter as simulation client, if we configure 200 threads (200 connections) to connect towards Tomcat Server 9.0.x (built from latest source code) embedded in our system and if we send 200 requests per second, we are getting the below two exceptions in Tomcat logs. 1.)Exception in thread "http-nio-x.y.z.a-1234-exec-4" java.lang.NullPointerException at org.apache.coyote.http2.Http2UpgradeHandler.endRequestBodyFrame(Http2UpgradeHandler.java:1416) at org.apache.coyote.http2.Http2AsyncUpgradeHandler.endRequestBodyFrame(Http2AsyncUpgradeHandler.java:39) at org.apache.coyote.http2.Http2Parser.readDataFrame(Http2Parser.java:207) at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:245) at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:163) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) 2.)Exception in thread "http-nio-x.y.z.a-1234-exec-13" java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206) at org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100) at org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401) at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:306) at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:163) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Build: The latest jars were built and embedded in our system from the Tomcat 9.0.x branch source . Resolution/Clarification requested: Can you please explain the root cause for this exception and fix the issue . Thanks in advance.
Please provide a test case. "Can you please explain the root cause for this exception and fix the issue . " This means you should investigate this in the user list first. You could have some concurrent operation recycling the request or something like that.
We got the exception while running the below case with message payload size 2kb Client - JMeter Memory : 1GB at jmeter side No of threads (users) : 200 Ram-Up period : 1 Loop count : 60 (or) can also be tried by stopping the client before receiving response
... and the application is, what, exactly? Can you get the Tomcat examples application to fail in this way with JMeter?
*** This bug has been marked as a duplicate of bug 64671 ***
Created attachment 37423 [details] JMX file
Created attachment 37424 [details] Jar file This is based of the example project https://github.com/jyeary/tomcat-9-embedded/ and modified to add HTTP2Upgrade protocol.Servlet can be found at http://localhost:1090/hello. java -cp <CLASSPATH> -jar tomcat-9-embedded-1.0.4.jar -Xmx1g <CLASSPATH> - where the latest tomcat jars are placed.
Hi Mark , The Null Pointer Exception and BufferOverflowException is from the jars built from the latest tomcat source code (9.0.x branch). It is not a duplicate of bug 64671. 1)I have attached the jmx file and jar to reproduce the BufferOverflowException. Steps to reproduce: *Client side : JMeter (attached Reproduction.jmx file) No of threads:200 Loop Count:10 (can be increased) Port:1080 *Server side (Embedded tomcat application : attached tomcat-9-embedded-1.0.4.jar) This is based of the example project https://github.com/jyeary/tomcat-9-embedded/ and modified to add HTTP2Upgrade protocol.Servlet can be found at http://localhost:1090/hello. java -cp <CLASSPATH> -jar tomcat-9-embedded-1.0.4.jar -Xmx1g <CLASSPATH> - where the latest tomcat jars are placed. 2)The Null pointer Exception is due to Null check not being added in the endRequestBodyFrame in Http2UpgradeHandler : public void endRequestBodyFrame(int streamId) throws Http2Exception { Stream stream = getStream(streamId, true); stream.getInputBuffer().onDataAvailable(); } Can you please check and help us with the fix. Thank you !
Correcting the servlet port number: http://localhost:1080/hello
If there is any update Please let us know .
"latest source code" is not specific. If you are going to build from source and then report a bug you need to provide the hash you build from.
No JMeter version specified. No mention that an additional plugin is required. No source provided for JAR file - it is very unlikely a Tomcat developer is going to download a random JAR from a bug report and run it. Despite the above, I have been able to reproduce this with some changes to the JMeter configuration to make requests against the examples web application that ships with Tomcat. I'm currently considering alternative ways to fix reduce the memory footprint of closed streams that doesn't involve nulling out references.
Fixed in: - master for 10.0.0-M8 onwards - 9.0.x for 9.0.38 onwards - 8.5.x for 8.5.58 onwards
Created attachment 37444 [details] JMX file and Source Code
Mark , Thank you for the response. Sorry to have missed out on a few points. I tried testing the same , post the fix with the hash https://github.com/apache/tomcat/commit/ea0a200fe29829bd88fb6fecb5549158896b68ea I can still see the BufferOverflow Exception.Please help us with this. JMeter Version: apache-jmeter-4.0 (with additional plugin for HTTP2) I have attached the source code and jmx file to reproduce the issue . To build the source code I have pointed to 9.0.36 in the pom.xml, but while executing the application , the classpath where the the tomcat jars built from the given hash is referred. java -cp <CLASSPATH> -jar tomcat-9-embedded-1.0.4.jar -Xmx1g <CLASSPATH> - where the latest tomcat (9.0.38) jars are placed.
I am unable to reproduce this with 9.0.x or 10.0.x. Ideally, we need a more reliable test case. Is it just the one stack trace you see with the BufferOverflowException? If there is more than one stack trace - or if other exceptions are present in the logs - it might be helpful to see each of them.
Hi Mark , It is only the BufferOverFlow stack trace that we get in our application. Also ,we have configured the maxThreads as 40 (in our application and the test source code that I had attached earlier) . Is maxThreads applicable for http2 as well ?
Any update on this please ? Thanks !
None. We need a test case that reliably reproduces the issue.
BufferOverflowException fixed in: Fixed in: - master for 10.0.0-M9 onwards - 9.0.x for 9.0.39 onwards 8.5.x is not affected.