Bug 50072

Summary: Blank responses
Product: Tomcat 6 Reporter: petr
Component: CatalinaAssignee: Tomcat Developers Mailing List <dev>
Severity: normal CC: knight, NOS
Priority: P2    
Version: 6.0.24   
Target Milestone: default   
Hardware: PC   
OS: Linux   
Attachments: Tcp dump log with blank and normal responses.
tcpdump log - nio connector
tcpdump log - bio connector
Proposed patch

Description petr 2010-10-11 02:00:58 UTC
Created attachment 26153 [details]
Tcp dump log with blank and normal responses.

We are facing problem with empty file response from server, after upgrading from Apache 6.0.20 to any higher version. Approx 20% generated responses went ok, 80% responses were empty files without any error log. Types of files was XML, HTML pages - with no diffrence. Files were requested by browsers, PHP scripts (XML) from different client servers.

I started to log all the requests on top of doGet() method and all the normal responses logged correctly. Incorrect requests never get to aplication, empty files had no log.

There was no diffrence when server had load on the resources were free. We use "org.apache.coyote.http11.Http11NioProtocol", requests were sent with GET method. I attach TCPdump from one of the client-side servers - what they see. After downgrade back to 6.0.20 the problem dissapeared, so we are kind of locked there.
Comment 1 Mark Thomas 2010-10-11 06:32:33 UTC
There haven't been any similar reports and 6.0.24 was released almost 9 months ago. Either the application is triggering a rare code-path in Tomcat which is hitting a bug or the application is sending invalid requests that Tomcat is now being stricter in how it is handling them.

I tried to read the attached trace in Wireshark but a dump of the text output from tcpdump is not easily imported. Looking through the text output I see some oddities (e.g. Host headers in a HTTP 1.0 request) but nothing that jumps out as obviously malformed.

Please provide a complete tcpdump trace of 5 to 10 requests in pcap format that demonstrates this issue.

Also, note there is no need to edit the file to obscure any host names, IP addresses etc. since they are all present in the hex data in the attached file.

You might also like to try running Tomcat with the BIO connector to see if the problem remains.
Comment 2 petr 2010-10-13 09:09:24 UTC
Created attachment 26168 [details]
tcpdump log - nio connector
Comment 3 petr 2010-10-13 09:09:59 UTC
Created attachment 26169 [details]
tcpdump log - bio connector
Comment 4 petr 2010-10-13 09:10:26 UTC
This problem has tomcat 6.0.24 and all next releases. I tested it now with version 6.0.29 and php on one of our client servers where the problem was detected before. Problem is with http 1.0 requests from php (fopen, simplexml_load_file functions) and NIO connector. Same script works fine with BIO connector. Http 1.1 requests (curl, fsockopen php functions) works fine with both connectors. I attach bio.log and nio.log tcpdump logs. Top +- 10 requests are from simplexml_load_file and next are curl and fsockopen.
Comment 5 Mark Thomas 2010-10-13 17:59:19 UTC
Created attachment 26173 [details]
Proposed patch
Comment 6 Mark Thomas 2010-10-13 18:02:49 UTC
Thanks for the additional traces and analysis. The analysis is particularly helpful. I tracked the problem down to a long standing issue in the NIO connector where request lines sent in more that one packet are likely to be mis-read. The simplest way to reproduce this is to use telnet.

I have applied a fix for this issue to 7.0.x and it will be included in 7.0.4 onwards.

I have proposed a port of the same fix for 6.0.x (patch attached).
Comment 7 petr 2010-10-19 04:14:37 UTC
Thank you. Works fine with your patch.
Comment 8 Mark Thomas 2010-10-19 08:29:54 UTC
Thanks for the feedback. Good to know the problem I found was the same one you were seeing.

Keeping open until the patch has been applied to 6.0.x
Comment 9 Mark Thomas 2010-11-03 15:07:55 UTC
Fixed in 6.0.x and will be included in 6.0.30 onwards.
Comment 10 Mark Thomas 2010-11-05 14:04:35 UTC
*** Bug 50224 has been marked as a duplicate of this bug. ***
Comment 11 Mark Thomas 2011-01-04 18:05:36 UTC
*** Bug 50505 has been marked as a duplicate of this bug. ***