Bug 48763 - AJP Connectors fail when POST requests don't have a Content-Length header
Summary: AJP Connectors fail when POST requests don't have a Content-Length header
Alias: None
Product: Tomcat Connectors
Classification: Unclassified
Component: isapi (show other bugs)
Version: 1.2.28
Hardware: PC Windows NT
: P2 major (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
Depends on:
Reported: 2010-02-18 00:34 UTC by Bruce G. Stewart
Modified: 2010-02-23 02:49 UTC (History)
1 user (show)

Read the first body chunk of an APR request that has a Transfer-Encoding header. (1.73 KB, application/octet-stream)
2010-02-18 14:07 UTC, Bruce G. Stewart

Note You need to log in before you can comment on or make changes to this bug.
Description Bruce G. Stewart 2010-02-18 00:34:25 UTC
This problem occurs with both the Java and APR connectors of the coyote AJP package. The symptom is, when an AJP connection is reused for multiple POST requests, sporadic errors occur on the second and subsequent requests.

The classes org.apache.coyote.ajp.AjpProcessor and org.apache.coyote.ajp.AjpAprProcessor look for a positive value in the Content-Length header to determine whether to read the first Ajp chunk of a request body.

The http spec says "The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer-Encoding header field in the request's header fields." Web service clients generated by Apache Axis2 are one source of POST requests that have no Content-Length header.

When a client sends a POST request with no Content-Length header, these classes emit a spurious GET_BODY_CHUNK request on the AJP connection. The first request on the AJP connection will usually succeed, but for subsequent requests over the same connection the protocol gets confused, sometimes processing headers as bodies, and so on.

One symptom of this that I have seen mentioned elsewhere is that web-service requests are rejected for bad UTF-8 characters.

A patch that worked for me was to change the doRead() method of the SocketInputBuffer inner-class, to check for the presence of Transfer-Encoding or Content-Length.

The problem is similar for the Java AJP and the APR AJP, and the same fix seems to work for both.
Comment 1 Mladen Turk 2010-02-18 05:06:08 UTC
Are there any chance to see the patch that fixed this for you?

Also which mod_jk you are using. Latest stable (1.2.28) should have
fixed C-L == 0 issues that can lead to the cross context data corruption.
However those fixes mostly address the missing body, not quite all the
C-L == 0 cases, so I'd like to see the patch that solves this.
Comment 2 Bruce G. Stewart 2010-02-18 14:07:12 UTC
Created attachment 25015 [details]
Read the first body chunk of an APR request that has a Transfer-Encoding header.

A proposed patch against tag/TOMCAT_6_0_24 is attached.
Comment 3 Mladen Turk 2010-02-18 14:20:27 UTC
OK, So the patch just executes if C-L is zero or not set but T-E is.
Seems reasonable. However I'd make sure the T-E is > 0 like we
are doing with C-L.

If you don't provide a complete patch I'll make something similar
in trunk, so we'll see weather it'll get backported, although I
see why it shouldn't cause it's not just a nice feature, it's a law :) 

Comment 4 Mladen Turk 2010-02-18 14:25:32 UTC
I meant T-E == chunked obviously :)
Comment 5 jfclere 2010-02-18 14:54:11 UTC
Do I miss something: In mod_proxy we have:
    /* read the first bloc of data */
    input_brigade = apr_brigade_create(p, r->connection->bucket_alloc);
    tenc = apr_table_get(r->headers_in, "Transfer-Encoding");
    if (tenc && (strcasecmp(tenc, "chunked") == 0)) {
        /* The AJP protocol does not want body data yet */
        ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
                     "proxy: request is chunked");
    } else {
The patch would just block and never get it no?
Comment 6 Remy Maucherat 2010-02-18 14:57:48 UTC
Before putting any fix, the question is if the first body message will be automatically sent (so Jean-Frédéric thinks it is only the case for c-l > 0, from what I understood).

All AJP implementations must have the same behavior, otherwise this is big trouble.
Comment 7 Mladen Turk 2010-02-18 15:04:10 UTC
Right, seems we'll need a descent test case when to issue a read.
However the major problem is that read is issued although not a
valid AJP sequence (no C-L header) and that should be rejected
by the AJP connector instead simply requesting a body chunk from
mod_proxy/mod_jk if Servlet thinks this is valid POST request.
Comment 8 Mladen Turk 2010-02-18 15:13:17 UTC
Are we supporting T-E chunked with the AJP at the first place.
Seems to me we don't have any filter inserted inside AJP like
we have for HTTP. If not, then T-E=chunked and C-L=0 should
be treated as invalid request and connection dropped to prevent
any further problems.
Comment 9 Bruce G. Stewart 2010-02-18 18:26:40 UTC
I am using the ISAPI_Redirect IIS plugin with IIS Version 7.

I see there is a separate build that offers "chunking transfer support". Would it solve my problem if I used that without patching Tomcat? Or make things worse? (I should probably go read the source and leave you guys alone.)

Thanks for your help and attention. I appreciate it.
Comment 10 Mladen Turk 2010-02-18 19:50:04 UTC
Isapi chunking is for sending the chunked data.
Won't help you there.
Anyhow your patch is wrong cause it just hides the symptoms.
The real problem is that client EOF is not handled correctly.
Comment 11 Bruce G. Stewart 2010-02-18 22:39:32 UTC
If the isapi_redirector is not supposed to send the first body block of such a message until it receives a get-body message, then the isapi_redirector must be faulty.

On the very first request, and all that follow, the first body block is available on the apr socket immediately after the header is read. With short messages, refillReadBuffer() is never called. With longer ones, the first read picks up a couple of hundred bytes, and refillReadBuffer gets the rest.

I haven't seen any problem related to the end of request data being mishandled.
Comment 12 Mladen Turk 2010-02-19 05:27:31 UTC
Special first body message is only send if C-L is present as this is defined by AJP protocol spec.
If it's there without C-L this is faulty and this should be fixed.
Comment 13 Mladen Turk 2010-02-19 10:59:47 UTC
Can you check with the latest 1.2.29 preview from

Use the isapi_redirect-1.2.29-dev-911726.dll
and rename to isapi_redirect.dll
There is also 64-bit version available at

1.2.29 has few fixes for tightening the AJP message sequence ordering
making it more robust for edge cases.
Comment 14 Bruce G. Stewart 2010-02-20 01:13:30 UTC
Mladen, isapi_redirect-1.2.29-dev-911726.dll (x64 version) acts just the same way - the first body block is sent right after the header.

Browsing the jk code, I see ajp_send_request() in jk_ajp_common.c includes this comment:

        /* || s->is_chunked - this can't be done here. The original protocol
           sends the first chunk of post data ( based on Content-Length ),
           and that's what the java side expects.
           Sending this data for chunked would break other ajp13 servers.

           Note that chunking will continue to work - using the normal read.

but code to test the value of s->is_chunked was removed between revisions 298092 and 298162.
Comment 15 Mladen Turk 2010-02-20 08:01:34 UTC
Right, it might be the case if IIS sets the ECB->cbTotalBytes
Seem we don't check if the C-L is present in that case which
might lead to initial post to be send even if the C-L is not
Give me till Monday and I'll create another snapshot with that
issues fixed so we make sure not to send the initial POST packet
if there is no C-L header

Anyhow can you try with Httpd and see are there any differences
since Httpd will not send the first packet.

Hope you'll be willing to test it.
Comment 16 Mladen Turk 2010-02-20 08:03:40 UTC
Changing to PC/NT since it seems only IIS is affected.
Also we should probably have IIS/Httpd/Netscape as additional fields
for filing the Connector bugs.
Comment 17 Mladen Turk 2010-02-21 07:24:15 UTC
Please check version isapi_redirect-1.2.29-dev-912307.dll from

It should have resolved initial POST data for T-E being send.
IIS sets 0xFFFFFFFF for cbTotalBytes which we were using for content-length
but this value means "Unknown body size".
This should also solve the posts larger then 4Gb

We are waiting on your input for making a final 1.2.29 :)
Comment 18 Bruce G. Stewart 2010-02-21 20:42:40 UTC
isapi_redirect-1.2.29-dev-912307.dll solves my issue.

(I saw the C-L -1 in the ajp messages to Tomcat, but since Tomcat looked for C-L > 0, I assumed that -1 was intentional. Sorry I didn't mention it - might have saved you some digging.)
Thank you very much.
Comment 19 Mladen Turk 2010-02-22 06:12:39 UTC
This is actually very important. Tomcat looks for C-L > 0
but inside JK we were just looking for C-L != 0 and send the
initial packet after the request.
Also we use 64-bits for C-L thus 0xFFFFFFFF mean 4Gb.
Now if the servlet didn't consume that packet there was a
chance it get interpreted as a new request. If it looked
as a normal request things could get pretty nasty.
The next request could receive the data from the previous request.
With the current fix instead sending that value we don't send the
C-L at all which according to the AJP spec mean "unknow".
Comment 20 Bruce G. Stewart 2010-02-22 18:27:32 UTC
This morning, I had some failures of another application using the .29 dll and the same Tomcat. I haven't yet determined if this is a misconfiguration issue or a bug, but I thought I should show it to you:

[Mon Feb 22 08:13:50.104 2010] [4320:3432] [info] init_jk::jk_isapi_plugin.c (2403): Starting Jakarta/ISAPI/isapi_redirector/1.2.29-dev-912307
[Mon Feb 22 08:13:50.151 2010] [4320:3432] [info] init_jk::jk_isapi_plugin.c (2573): Jakarta/ISAPI/isapi_redirector/1.2.29-dev-912307 initialized
[Mon Feb 22 08:13:57.530 2010] [4320:3432] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (1264): wrong message size 8196 8192 from
[Mon Feb 22 08:13:57.530 2010] [4320:1896] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (1264): wrong message size 8196 8192 from
[Mon Feb 22 08:13:57.546 2010] [4320:3432] [error] ajp_get_reply::jk_ajp_common.c (2072): (ajp13w) Tomcat is down or network problems. Part of the response has already been sent to the client

[... more of the same, and eventually ...]

[Mon Feb 22 08:13:58.107 2010] [4320:3432] [info] service::jk_lb_worker.c (1464): All tomcat instances are busy or in error state
[Mon Feb 22 08:13:58.107 2010] [4320:3432] [error] service::jk_lb_worker.c (1469): All tomcat instances failed, no more workers left
Comment 21 Mladen Turk 2010-02-22 18:38:23 UTC
> ajp_connection_tcp_get_message::jk_ajp_common.c (1264): wrong message size 8196

This looks like mismatch in maxPacketSize. Seems the Tomcat is configured
for larger then default packet sizes.
Comment 22 Rainer Jung 2010-02-23 02:49:31 UTC
Will be part of 1.2.29.