Created attachment 28258 [details]
Tools for reproducing the problem - script, test load, config, backend JSP

When request bodies are posted with gzip compression + chunked transfer, then proxied to a Tomcat backend, they are subject to sporadic failures with all three proxy methods I've tried: mod_proxy_http, mod_proxy_ajp, and mod_jk.

The problem does not occur without gzip. The failures are similar enough that I suspect the problem has to do with the hand-off from mod_deflate, though it could be further along in the request handling. It does not succeed or fail the same each request even when exactly the same content is sent with exactly the same chunk sizes, so I suspect that network timing and/or internal state may play a role.

This may be related to https://issues.apache.org/bugzilla/show_bug.cgi?id=52492 - then again, it may not. It seemed different enough to me to merit a separate issue.

This problem came from a real-world system issue that initially I suspected was an AJP problem. When transferred via mod_jk, the request body would sporadically be truncated when it was received by the backend (as confirmed by packet trace). Aside from that all appears fine.

With mod_proxy_ajp, this use case never works at all that I have seen. The entire body may be transmitted, but instead of sending the finishing 0-size chunk, httpd closes the proxy connection and returns a 200 to the client (with empty body). Errors in the log indicate mod_proxy_ajp.c(382): (20014)Internal error: ap_get_brigade failed

With mod_proxy_http, this manifests as a 0-size chunk coming in the middle of the request transfer. The backend interprets this as the end of the request (which is normally what the 0-size chunk means) and responds normally to that truncated request. Then when the proxy continues sending more chunks, the backend returns a 400 and closes the connection. The response to the client depends on timing - either the 200 response from the backend is returned, or (if the connection is closed before the request is fully sent) a 502 error is returned.

Side note: mod_proxy_http passes along the Content-Encoding: gzip header to the backend even though the traffic is not compressed at that point. This is probably a bug and might cause problems with some backends.

In order to test all this, I put together a Perl script that can gzip and chunk a request allowing me to control chunk sizes (I found that some patterns made this occur more often than others). I also created a JSP for the backend that counts the incoming bytes and returns the count in the response so that the script can verify whether the full content made it through. The content is arbitrary, but for my tests, I chose a small portion of the system dictionary. These are all attached. 

I will further attach logs and packet traces recording failures with each proxy method. These are against 2.2.21, but I have observed the same results with other versions (2.2.22, 2.2.17, 2.3.16). Again, these can be very sporadic, so when reproducing, many requests may be required before anything goes wrong (the script automates this). You may see many successes or many failures in a row. It would not surprise me if this varied by OS, libz version, architecture, etc. but I know it's not specific to my test system. In general, a good way to trigger the problem is the "randub" algorithm with a small chunk size (like 100 to send chunks with random sizes between 100 and 200).

My tests have been on Linux (CentOS 5).
$ uname -a
Linux 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:32:29 EDT 2011 i686 i686 i386 GNU/Linux
$ rpm -q zlib
zlib-1.2.3-4.el5

Thanks for taking a look at this.