Bug 43846 - Race condition with NIO connector parsing chunked response
Summary: Race condition with NIO connector parsing chunked response
Alias: None
Product: Tomcat 6
Classification: Unclassified
Component: Connectors (show other bugs)
Version: 6.0.14
Hardware: All All
: P2 regression (vote)
Target Milestone: default
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2007-11-12 09:08 UTC by Guy Molinari
Modified: 2008-01-18 06:04 UTC (History)
1 user (show)

Jython/Grinder script (1.77 KB, application/octet-stream)
2007-11-12 09:14 UTC, Guy Molinari
Grinder properties file (1.12 KB, application/octet-stream)
2007-11-12 09:16 UTC, Guy Molinari
Reponse generator servlet (1.49 KB, application/octet-stream)
2007-11-12 09:20 UTC, Guy Molinari
Use this Jython script instead (1.70 KB, application/octet-stream)
2007-11-13 11:46 UTC, Guy Molinari

Note You need to log in before you can comment on or make changes to this bug.
Description Guy Molinari 2007-11-12 09:08:54 UTC
The problem seems to relate to parsing the body content when the **Transfer-
Encoding: chunked** header is present.  It appears that the chunk length gets
corrupted under load and the client is unable to parse the chunk out of the 
response body.

When the NIO connector parameter **socket.appWriteBufSize** is set to a value 
larger than the total response body, the error condition does not occur.

One might succesfully argue that this is proper performance tuning.  The Tomcat 
documents point out that to scale a large number of long held connections, the 
buffer sizes may need to be less than the response body (the memory footprint 
would be quite large otherwise).

Could there be a race condition involving the response buffering code?  

I have confirmed this behavior on both JDKs 1.5 and 1.6 on both Windows 2003 
and Linux.

Apache Bench does not appear to fully parse the response so it will not be 
helpful in reproducing the issue.   The Grinder framework does.  I am including 
the Grinder scripts so that the issue may be reproduced.
Comment 1 Guy Molinari 2007-11-12 09:14:10 UTC
Created attachment 21114 [details]
Jython/Grinder script

Jython script referenced from grinder properties file.	 This script does most
of the interaction with the Tomcat server.  It is configured via a grinder
properties file.
Comment 2 Guy Molinari 2007-11-12 09:16:30 UTC
Created attachment 21115 [details]
Grinder properties file

Use this file as a starting point for your Grinder setup.  This defines the
number of threads, etc. to configure the test client.
Comment 3 Guy Molinari 2007-11-12 09:20:56 UTC
Created attachment 21116 [details]
Reponse generator servlet

This is a very simple Java servlet.  It generates a reponse body >8K.  This
somewhat larger response seems to render with Transfer-Encoding: chunked rather
than a Content-Length header.  This is what we want because the smaller
response bodies seem to work OK.  In the real world responses larger than 8K
are the norm.
Comment 4 Filip Hanik 2007-11-12 10:53:05 UTC
Thank you, I will take a look.
Comment 5 Filip Hanik 2007-11-13 11:41:23 UTC
I'm unable to get your grinder scripts to work.
There are several errors that happen, for one, random is not available on a
clean download of grinder.

what would be helpful, is if you could create a test case, that I can run
without trying to figure out grinder (sorry I'm a newbie to this) for most of a day.

Comment 6 Guy Molinari 2007-11-13 11:46:35 UTC
Created attachment 21123 [details]
Use this Jython script instead

Use this script instead of the first one attached.   Sorry for the trouble
Comment 7 Guy Molinari 2007-11-13 11:49:21 UTC
My apolgies Filip.   I've attached an updated Jython script.  This one does not 
depend on random.
Comment 8 Filip Hanik 2007-11-13 14:00:46 UTC
I'm able to reproduce the problem in the connector using a JMeter script.
The error doesn't exist in the previous trunk connector, now in sandbox

There are features in that branch not present in current release. I will look
into the option of porting the NIO enhancements from that branch to 6.0.x

Comment 9 Filip Hanik 2007-11-13 15:28:13 UTC
I've created a patch for 6.0.x branch (a port from the sandbox)

feel free to try it out, and let me know the feedback

Comment 10 Guy Molinari 2007-11-14 12:10:26 UTC
I've run a series of tests of various kinds and it appears that this issue is 
fixed in the patched version.   Would this fix going into a 6.0.15 release?
Comment 11 Filip Hanik 2007-11-19 20:19:58 UTC
Patch has been added to 6.0.x and will go out with 6.0.16 next release.
Thanks for the feedback.