|Summary:||Race condition with NIO connector parsing chunked response|
|Product:||Tomcat 6||Reporter:||Guy Molinari <guy.molinari>|
|Component:||Connectors||Assignee:||Tomcat Developers Mailing List <dev>|
Grinder properties file
Reponse generator servlet
Use this Jython script instead
Description Guy Molinari 2007-11-12 09:08:54 UTC
The problem seems to relate to parsing the body content when the **Transfer- Encoding: chunked** header is present. It appears that the chunk length gets corrupted under load and the client is unable to parse the chunk out of the response body. When the NIO connector parameter **socket.appWriteBufSize** is set to a value larger than the total response body, the error condition does not occur. One might succesfully argue that this is proper performance tuning. The Tomcat documents point out that to scale a large number of long held connections, the buffer sizes may need to be less than the response body (the memory footprint would be quite large otherwise). Could there be a race condition involving the response buffering code? I have confirmed this behavior on both JDKs 1.5 and 1.6 on both Windows 2003 and Linux. Apache Bench does not appear to fully parse the response so it will not be helpful in reproducing the issue. The Grinder framework does. I am including the Grinder scripts so that the issue may be reproduced.
Comment 1 Guy Molinari 2007-11-12 09:14:10 UTC
Created attachment 21114 [details] Jython/Grinder script Jython script referenced from grinder properties file. This script does most of the interaction with the Tomcat server. It is configured via a grinder properties file.
Comment 2 Guy Molinari 2007-11-12 09:16:30 UTC
Created attachment 21115 [details] Grinder properties file Use this file as a starting point for your Grinder setup. This defines the number of threads, etc. to configure the test client.
Comment 3 Guy Molinari 2007-11-12 09:20:56 UTC
Created attachment 21116 [details] Reponse generator servlet This is a very simple Java servlet. It generates a reponse body >8K. This somewhat larger response seems to render with Transfer-Encoding: chunked rather than a Content-Length header. This is what we want because the smaller response bodies seem to work OK. In the real world responses larger than 8K are the norm.
Comment 4 Filip Hanik 2007-11-12 10:53:05 UTC
Thank you, I will take a look. Filip
Comment 5 Filip Hanik 2007-11-13 11:41:23 UTC
I'm unable to get your grinder scripts to work. There are several errors that happen, for one, random is not available on a clean download of grinder. what would be helpful, is if you could create a test case, that I can run without trying to figure out grinder (sorry I'm a newbie to this) for most of a day. thanks Filip
Comment 6 Guy Molinari 2007-11-13 11:46:35 UTC
Created attachment 21123 [details] Use this Jython script instead Use this script instead of the first one attached. Sorry for the trouble Filip.
Comment 7 Guy Molinari 2007-11-13 11:49:21 UTC
My apolgies Filip. I've attached an updated Jython script. This one does not depend on random.
Comment 8 Filip Hanik 2007-11-13 14:00:46 UTC
I'm able to reproduce the problem in the connector using a JMeter script. The error doesn't exist in the previous trunk connector, now in sandbox http://svn.apache.org/viewvc/tomcat/sandbox/gdev6x/ There are features in that branch not present in current release. I will look into the option of porting the NIO enhancements from that branch to 6.0.x Filip
Comment 9 Filip Hanik 2007-11-13 15:28:13 UTC
I've created a patch for 6.0.x branch (a port from the sandbox) http://people.apache.org/~fhanik/patches/apache-tomcat-6.0.x-niofix.zip feel free to try it out, and let me know the feedback Filip
Comment 10 Guy Molinari 2007-11-14 12:10:26 UTC
I've run a series of tests of various kinds and it appears that this issue is fixed in the patched version. Would this fix going into a 6.0.15 release?
Comment 11 Filip Hanik 2007-11-19 20:19:58 UTC
Patch has been added to 6.0.x and will go out with 6.0.16 next release. Thanks for the feedback.