Bug 43846

Summary:	Race condition with NIO connector parsing chunked response
Product:	Tomcat 6	Reporter:	Guy Molinari <guy.molinari>
Component:	Connectors	Assignee:	Tomcat Developers Mailing List <dev>
Status:	RESOLVED FIXED
Severity:	regression	CC:	cpierret
Priority:	P2
Version:	6.0.14
Target Milestone:	default
Hardware:	All
OS:	All
Attachments:	Jython/Grinder script Grinder properties file Reponse generator servlet Use this Jython script instead

Description Guy Molinari 2007-11-12 09:08:54 UTC

The problem seems to relate to parsing the body content when the **Transfer-
Encoding: chunked** header is present.  It appears that the chunk length gets
corrupted under load and the client is unable to parse the chunk out of the 
response body.

When the NIO connector parameter **socket.appWriteBufSize** is set to a value 
larger than the total response body, the error condition does not occur.

One might succesfully argue that this is proper performance tuning.  The Tomcat 
documents point out that to scale a large number of long held connections, the 
buffer sizes may need to be less than the response body (the memory footprint 
would be quite large otherwise).

Could there be a race condition involving the response buffering code?  

I have confirmed this behavior on both JDKs 1.5 and 1.6 on both Windows 2003 
and Linux.

Apache Bench does not appear to fully parse the response so it will not be 
helpful in reproducing the issue.   The Grinder framework does.  I am including 
the Grinder scripts so that the issue may be reproduced.

Comment 1 Guy Molinari 2007-11-12 09:14:10 UTC

Created attachment 21114 [details]
Jython/Grinder script

Jython script referenced from grinder properties file.	 This script does most
of the interaction with the Tomcat server.  It is configured via a grinder
properties file.

Comment 2 Guy Molinari 2007-11-12 09:16:30 UTC

Created attachment 21115 [details]
Grinder properties file

Use this file as a starting point for your Grinder setup.  This defines the
number of threads, etc. to configure the test client.

Comment 3 Guy Molinari 2007-11-12 09:20:56 UTC

Created attachment 21116 [details]
Reponse generator servlet

This is a very simple Java servlet.  It generates a reponse body >8K.  This
somewhat larger response seems to render with Transfer-Encoding: chunked rather
than a Content-Length header.  This is what we want because the smaller
response bodies seem to work OK.  In the real world responses larger than 8K
are the norm.

Comment 4 Filip Hanik 2007-11-12 10:53:05 UTC

Thank you, I will take a look.
Filip

Comment 5 Filip Hanik 2007-11-13 11:41:23 UTC

I'm unable to get your grinder scripts to work.
There are several errors that happen, for one, random is not available on a
clean download of grinder.

what would be helpful, is if you could create a test case, that I can run
without trying to figure out grinder (sorry I'm a newbie to this) for most of a day.

thanks
Filip

Comment 6 Guy Molinari 2007-11-13 11:46:35 UTC

Created attachment 21123 [details]
Use this Jython script instead

Use this script instead of the first one attached.   Sorry for the trouble
Filip.

Comment 7 Guy Molinari 2007-11-13 11:49:21 UTC

My apolgies Filip.   I've attached an updated Jython script.  This one does not 
depend on random.

Comment 8 Filip Hanik 2007-11-13 14:00:46 UTC

I'm able to reproduce the problem in the connector using a JMeter script.
The error doesn't exist in the previous trunk connector, now in sandbox
http://svn.apache.org/viewvc/tomcat/sandbox/gdev6x/

There are features in that branch not present in current release. I will look
into the option of porting the NIO enhancements from that branch to 6.0.x

Filip

Comment 9 Filip Hanik 2007-11-13 15:28:13 UTC

I've created a patch for 6.0.x branch (a port from the sandbox)
http://people.apache.org/~fhanik/patches/apache-tomcat-6.0.x-niofix.zip

feel free to try it out, and let me know the feedback

Filip

Comment 10 Guy Molinari 2007-11-14 12:10:26 UTC

I've run a series of tests of various kinds and it appears that this issue is 
fixed in the patched version.   Would this fix going into a 6.0.15 release?

Comment 11 Filip Hanik 2007-11-19 20:19:58 UTC

Patch has been added to 6.0.x and will go out with 6.0.16 next release.
Thanks for the feedback.