Bug 50975 - IIS connector times out on Transfer Encoded content, never sending the chunked content
Summary: IIS connector times out on Transfer Encoded content, never sending the chunke...
Alias: None
Product: Tomcat Connectors
Classification: Unclassified
Component: isapi (show other bugs)
Version: unspecified
Hardware: PC All
: P2 major (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
: 48940 (view as bug list)
Depends on:
Reported: 2011-03-25 18:46 UTC by Aaron Johnson
Modified: 2011-04-11 05:39 UTC (History)
1 user (show)

Zip of logs and network traces (8.79 KB, application/zip)
2011-03-25 18:46 UTC, Aaron Johnson
Don't send content length for chunked content (1.68 KB, patch)
2011-03-28 18:05 UTC, Aaron Johnson
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aaron Johnson 2011-03-25 18:46:50 UTC
Created attachment 26799 [details]
Zip of logs and network traces

We have a client application making calls through web services that are using Transfer-Encoding: chunked.  This scenario hasn't ever worked possibly due to bug 48940 and bug 48763.  It works fine against a wide variety of other setups (JBoss/WebSphere/WebLogic/Apache/IBM IHS/iPlanet) that don't include the IIS connector.  In the isapi_redirect.properties file I have enable_chunked_encoding=true -- though not sure it matters post 1.2.28.  Here are the symptoms:

1.2.27 (chunked and non chunked versions) - Some POST/T-E=chunked requests work, others fail.
1.2.28 (chunked and non chunked versions) - Some POST/T-E=chunked requests work, others fail.
1.2.30 - Read times out on the client waiting for a response
1.2.31 - Read times out on the client waiting for a response

I'm attributing the failure of 1.2.27/28 on already known/fixed bugs and focusing on the behavior for 1.2.31.

Network tracing between both the client and connector and the connector and JBoss 4.3.0 indicate that the client is sending a complete request and that the connector is still waiting for something else from the client, or has received it but isn't sending the chunked content to JBoss.

Specifically of interest is that the connector is sending:
REQ:POST /rm/services/ServerInfo.asmx
But never sends the chunked content

The attachment has two sets of files for two scenarios, a WireShark network trace and the isapi_redirect.log.

The client_to_connector scenario was on Windows 7 x64 / IIS 7.5.7600.16385 / isapi_redirect-1.2.31.dll for amd64

The connector_to_jboss scenario was on Windows Server 2008 x64 / IIS 7.5.7600.16385 / isapi_redirect-1.2.31.dll for amd64
Comment 1 Aaron Johnson 2011-03-28 18:05:44 UTC
Created attachment 26808 [details]
Don't send content length for chunked content

From what I can tell, in the case of chunking, the headers are sent first then the chunked content.  What seems to be happening is that when both Transfer-Encoding=chunked and content-type=nnnn set the connector endpoint never requests the body because it still thinks there is content in the message.  This fix simply removes the content-type field if transfer-encoding=chunked or content > 4GB -- basically just moving the logic around.

I tried to figure out what part of IIS is setting the content-type, but never came to any conclusions.

Is there any way to get a 64-bit built binary to test out these changes for a customer -- I only had the ability to build a 32-bit binary?
Comment 2 Aaron Johnson 2011-03-28 18:07:35 UTC
Comment on attachment 26808 [details]
Don't send content length for chunked content

I mistakenly referred to content type, when I should have said content-length in the comment above.
Comment 3 Tim Whittington 2011-03-31 03:59:11 UTC
Hi Aaron

Thanks for the report (and providing in logs and network traces).

Looking at the traces you supplied, it doesn't look like there is a Content-Length in the request from the client -> IIS/connector, so I'm confused by the patch modifying the behaviour when the Content-Length is present in the request.


POST /rm/services/ServerInfo.asmx HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://www.logiclibrary.com/types/getServerSoapVersion"
User-Agent: Axis/1.3
Transfer-Encoding: chunked

I think there are a few 'quirky' parts in the code around this, but I'd like to be definite on the failing test case before delving in.
Comment 4 Aaron Johnson 2011-03-31 10:10:43 UTC
The problem is that IIS is inserting the content length into the chunked request at some point in its filtering chain.  You are correct the client isn't sending content-length, but when the isapi_redirect.dll connector gets the request it does contain the C-L header.  I think that is what is causing the connector endpoint to hang.  The C-L header is confusing the endpoint because it is waiting for the chunked content because of the C-L header, but the connector is expecting the endpoint to request the chunked content.

I'd suspect that maybe the other connectors would have this bug as well, but it is only IIS putting C-L header on causing both the C-L and C-E=chunked headers to appear on the request.

I was able to test out the patch on 64-bit IIS and it does solve my immediate problem with the POSTs hanging, but I'm not familiar with what the specs say about having both C-L and C-E on the same request.
Comment 5 Tim Whittington 2011-04-03 19:27:26 UTC
The HTTP spec is clear on the desired behaviour here:

"    If a Content-Length header field (section 14.13) is present, its
     decimal value in OCTETs represents both the entity-length and the
     transfer-length. The Content-Length header field MUST NOT be sent
     if these two lengths are different (i.e., if a Transfer-Encoding

     header field is present). If a message is received with both a
     Transfer-Encoding header field and a Content-Length header field,
     the latter MUST be ignored.
Comment 6 Tim Whittington 2011-04-11 04:17:53 UTC
*** Bug 48940 has been marked as a duplicate of this bug. ***
Comment 7 Tim Whittington 2011-04-11 05:39:19 UTC
I've committed a fix similar to the one provided - thanks for providing a patch.
Bug 48940 reported what I think is the same issue, with a similar analysis.

I can't replicate the actual error case in my systems, although the analysis seems fairly conclusive, and the fix implemented is correct wrt interpretation of the HTTP RFC in any case.
I've verified everything works fine if the Content-Length is (invalidly) included in the original request to IIS - e.g. the header is dropped before forwarding to Tomcat.

Fix as implemented will be in 1.2.32.
If anyone can reproduce the issue and is able to test, that'd be appreciated.