Bug 64710 - NullPointerException in Http2UpgradeHandler.endRequestBodyFrame and BufferOverflowException in SocketBufferHandler
Summary: NullPointerException in Http2UpgradeHandler.endRequestBodyFrame and BufferOve...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 9.0.x
Hardware: HP Linux
: P2 regression (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-04 10:11 UTC by Arshiya
Modified: 2020-10-01 10:25 UTC (History)
0 users



Attachments
JMX file (48.91 KB, application/xml)
2020-09-06 10:43 UTC, Arshiya
Details
Jar file (5.23 KB, application/zip)
2020-09-06 10:44 UTC, Arshiya
Details
JMX file and Source Code (15.33 KB, application/x-zip-compressed)
2020-09-09 14:01 UTC, Arshiya
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Arshiya 2020-09-04 10:11:42 UTC
Sub-Component - Coyote

OS : Redhat Linux

Overview:

Tomcat implemented to transport http/2 packets between 2 systems (h2c connection). 

Steps to Reproduce:

With JMeter as simulation client, if we configure 200 threads (200 connections) to connect towards Tomcat Server 9.0.x (built from latest source code) embedded in our system and if we send 200 requests per second, we are getting the below two exceptions in Tomcat logs. 

    1.)Exception in thread "http-nio-x.y.z.a-1234-exec-4" java.lang.NullPointerException 
        at org.apache.coyote.http2.Http2UpgradeHandler.endRequestBodyFrame(Http2UpgradeHandler.java:1416)
        at org.apache.coyote.http2.Http2AsyncUpgradeHandler.endRequestBodyFrame(Http2AsyncUpgradeHandler.java:39)
        at org.apache.coyote.http2.Http2Parser.readDataFrame(Http2Parser.java:207)
        at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:245)
        at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:163)
        at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
		
	2.)Exception in thread "http-nio-x.y.z.a-1234-exec-13" java.nio.BufferOverflowException
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206)
        at org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
        at org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
        at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:306)
        at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:163)
        at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)


Build:

The latest jars were built and embedded in our system from the Tomcat 9.0.x branch source .

Resolution/Clarification requested:
Can you please explain the root cause for this exception and fix the issue . 

Thanks in advance.
Comment 1 Remy Maucherat 2020-09-04 10:25:55 UTC
Please provide a test case.

"Can you please explain the root cause for this exception and fix the issue . "
This means you should investigate this in the user list first. You could have some concurrent operation recycling the request or something like that.
Comment 2 Arshiya 2020-09-04 11:56:10 UTC
We got the exception while running the below case with message payload size 2kb
Client - JMeter
Memory : 1GB at jmeter side
No of threads (users) : 200 
Ram-Up period         : 1 
Loop count            : 60

(or) can also be tried by stopping the client before receiving response
Comment 3 Christopher Schultz 2020-09-04 16:17:28 UTC
... and the application is, what, exactly? Can you get the Tomcat examples application to fail in this way with JMeter?
Comment 4 Mark Thomas 2020-09-05 15:55:00 UTC

*** This bug has been marked as a duplicate of bug 64671 ***
Comment 5 Arshiya 2020-09-06 10:43:46 UTC
Created attachment 37423 [details]
JMX file
Comment 6 Arshiya 2020-09-06 10:44:46 UTC
Created attachment 37424 [details]
Jar file

This is based of the example project https://github.com/jyeary/tomcat-9-embedded/ and modified to add HTTP2Upgrade protocol.Servlet can be found at http://localhost:1090/hello.
java -cp <CLASSPATH> -jar tomcat-9-embedded-1.0.4.jar -Xmx1g
<CLASSPATH> - where the latest tomcat jars are placed.
Comment 7 Arshiya 2020-09-06 10:46:50 UTC
Hi Mark , 

The Null Pointer Exception and BufferOverflowException is from the jars built from the latest tomcat source code (9.0.x branch). It is not a duplicate of bug 64671.

1)I have attached the jmx file and jar to reproduce the BufferOverflowException.
Steps to reproduce:
*Client side : JMeter (attached Reproduction.jmx file)
No of threads:200
Loop Count:10 (can be increased)
Port:1080
*Server side (Embedded tomcat application : attached tomcat-9-embedded-1.0.4.jar)
This is based of the example project https://github.com/jyeary/tomcat-9-embedded/ and modified to add HTTP2Upgrade protocol.Servlet can be found at http://localhost:1090/hello.
java -cp <CLASSPATH> -jar tomcat-9-embedded-1.0.4.jar -Xmx1g
<CLASSPATH> - where the latest tomcat jars are placed.

2)The Null pointer Exception is due to Null check not being added in the endRequestBodyFrame in Http2UpgradeHandler :

    public void endRequestBodyFrame(int streamId)
        throws Http2Exception
    {
        Stream stream = getStream(streamId, true);
        stream.getInputBuffer().onDataAvailable();
    }

Can you please check and help us with the fix.
Thank you !
Comment 8 Arshiya 2020-09-06 11:56:47 UTC
Correcting the servlet port number: http://localhost:1080/hello
Comment 9 Arshiya 2020-09-07 16:28:59 UTC
If there is any update Please let us  know .
Comment 10 Mark Thomas 2020-09-07 22:27:07 UTC
"latest source code" is not specific. If you are going to build from source and then report a bug you need to provide the hash you build from.
Comment 11 Mark Thomas 2020-09-07 23:08:11 UTC
No JMeter version specified.
No mention that an additional plugin is required.
No source provided for JAR file - it is very unlikely a Tomcat developer is going to download a random JAR from a bug report and run it.

Despite the above, I have been able to reproduce this with some changes to the JMeter configuration to make requests against the examples web application that ships with Tomcat.

I'm currently considering alternative ways to fix reduce the memory footprint of closed streams that doesn't involve nulling out references.
Comment 12 Mark Thomas 2020-09-08 09:53:55 UTC
Fixed in:
- master for 10.0.0-M8 onwards
- 9.0.x for 9.0.38 onwards
- 8.5.x for 8.5.58 onwards
Comment 13 Arshiya 2020-09-09 14:01:19 UTC
Created attachment 37444 [details]
JMX file and Source Code
Comment 14 Arshiya 2020-09-09 14:02:27 UTC
Mark , Thank you for the response.
Sorry to have missed out on a few points.

I tried testing the same , post the fix with the hash
https://github.com/apache/tomcat/commit/ea0a200fe29829bd88fb6fecb5549158896b68ea 

I can still see the BufferOverflow Exception.Please help us with this.

JMeter Version: apache-jmeter-4.0 (with additional plugin for HTTP2)
I have attached the source code and jmx file to reproduce the issue .
To build the source code I have pointed to 9.0.36 in the pom.xml, but while executing the application , the classpath where the the tomcat jars built from the given hash is referred.

java -cp <CLASSPATH> -jar tomcat-9-embedded-1.0.4.jar -Xmx1g
<CLASSPATH> - where the latest tomcat (9.0.38) jars are placed.
Comment 15 Mark Thomas 2020-09-09 17:28:04 UTC
I am unable to reproduce this with 9.0.x or 10.0.x.

Ideally, we need a more reliable test case.

Is it just the one stack trace you see with the BufferOverflowException? If there is more than one stack trace - or if other exceptions are present in the logs - it might be helpful to see each of them.
Comment 16 Arshiya 2020-09-10 12:29:24 UTC
Hi Mark ,
It is only the BufferOverFlow stack trace that we get in our application.

Also ,we have configured the maxThreads as 40 (in our application and the test source code that I had attached earlier) .

Is maxThreads applicable for http2 as well ?
Comment 17 Arshiya 2020-09-17 04:15:06 UTC
Any update on this please ?

Thanks !
Comment 18 Mark Thomas 2020-09-17 07:56:52 UTC
None. We need a test case that reliably reproduces the issue.
Comment 19 Mark Thomas 2020-10-01 10:25:36 UTC
BufferOverflowException fixed in:

Fixed in:
- master for 10.0.0-M9 onwards
- 9.0.x for 9.0.39 onwards

8.5.x is not affected.