Created attachment 37509 [details]
JMX file

Sub-Component - Coyote

OS : Redhat Linux

Overview:

Embedded Tomcat version 9.0.39 is implemented to transport http/2 packets between 2 systems (h2c connection).
Http2 packets of varied payload size from the client are processed asynchronously .
The content-type of request/response is application/json. We see that there are many exceptions that are printed when trying to parse the input JSON request , though the response is sent successfully . 
On printing the JSON along with the exception we see that the :
1) The input JSON received is incomplete.
2) The input JSON is not received as it was sent (i.e data seems to be jumbled).
3) The input JSON is empty.

In production we use embedded tomcat version 9.0.38 and see that these payload related exceptions are flooded in the logs with around 300 connections and a payload size of upto 34KB.
We have tried and reproduced these exceptions with a sample code implementing embedded tomcat version 9.0.39 with payload size of 55KB. The same request is sent at tps of 20 , but the exception occurs for few request at random . Please find below the steps to reproduce. I have attached the sample code / input JMX file to reproduce the issue. 
 
Steps to Reproduce:

With JMeter as simulation client, on configuring 700 threads (700 connections) to connect towards Tomcat Server 9.0.39 embedded in our system and on sending 20 requests per second with payload of 55KB and letting the test run in an infinite loop , we see the above mentioned exceptions printed at random times (within 15 minutes of running the test).

Client:
JMeter 5.3 with additional HTTP2 sampler.
No of threads: 700
Ramp-up period:10 seconds	
Loop:Infinite
Payload size: around 55KB
Constant Throughput Timer added to limit the tps to 20.
Random Variable Generator added to the JSON request to uniquely identify for which request the exception is printed and to map it in the PCAP collected .

SERVER:
Sample application attached .
MaxThreads configured in tomcat is 200(all other parameters are the tomcat defaults).
The input requests are processed asynchronously with 40 threads.

Exceptions printed from the sample code:
1)Payload is empty
2)Exception in thread "pool-1-thread-20" java.lang.RuntimeException: Failed to convert JSON String to JSON Object for{
   "id":999312437,
   ..
   ..
   ..
      "batters128":{
      "batter":[
         {
            "id":"1001",
            "type":"Regular"
         },
         {
            "id":"1002",
            "type":"Chocolate"
         },
         {
            "id"

	at com.bluelotussoftware.tomcat.embedded.Task.run(Task.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.json.JSONException: Expected a ':' after a key at 46585 [character 0 line 2570]
    at org.json.JSONTokener.syntaxError(JSONTokener.java:507)
	at org.json.JSONObject.<init>(JSONObject.java:240)
	at org.json.JSONTokener.nextValue(JSONTokener.java:431)
	at org.json.JSONArray.<init>(JSONArray.java:125)
	at org.json.JSONTokener.nextValue(JSONTokener.java:434)
	at org.json.JSONObject.<init>(JSONObject.java:252)
	at org.json.JSONTokener.nextValue(JSONTokener.java:431)
	at org.json.JSONObject.<init>(JSONObject.java:252)
	at org.json.JSONObject.<init>(JSONObject.java:406)
	at com.bluelotussoftware.tomcat.embedded.Task.run(Task.java:43)
	... 3 more
	
The exceptions are printed along with the input JSON ( with the unique random number) and mapped in the PCAP by applying filter "json.value.number == 999312437" . I have attached the PCAP for the above mentioned case for which the payload exception was printed and then the request was processed successfully.
For case "Payload is Empty" we are not able to map the exact request.
We want to understand why the exception is printed stating data is incomplete / empty though the data is received completely and processed as viewed in the pcap .

Build:
Embedded tomcat 9.0.39

Resolution/Clarification requested:
Can you please explain the root cause for these exceptions and fix the issue . 

Thanks in advance.