Bug 52595 - requests with gzip+chunked encoded body don't proxy reliably
Summary: requests with gzip+chunked encoded body don't proxy reliably
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_deflate (show other bugs)
Version: 2.2.21
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Keywords: MassUpdate
Depends on:
Reported: 2012-02-03 19:57 UTC by Luke Meyer
Modified: 2018-11-07 21:09 UTC (History)
2 users (show)

Tools for reproducing the problem - script, test load, config, backend JSP (36.25 KB, application/x-gzip)
2012-02-03 19:57 UTC, Luke Meyer
Test case and results for mod_jk method (only one success and failure from the run are included in the trace, to keep size down) (139.53 KB, application/x-gzip)
2012-02-03 20:07 UTC, Luke Meyer
Test case and results for mod_proxy_http method (141.26 KB, application/x-gzip)
2012-02-03 20:08 UTC, Luke Meyer
Test case and results for mod_proxy_ajp method (143.67 KB, application/x-gzip)
2012-02-03 20:08 UTC, Luke Meyer

Note You need to log in before you can comment on or make changes to this bug.
Description Luke Meyer 2012-02-03 19:57:33 UTC
Created attachment 28258 [details]
Tools for reproducing the problem - script, test load, config, backend JSP

When request bodies are posted with gzip compression + chunked transfer, then proxied to a Tomcat backend, they are subject to sporadic failures with all three proxy methods I've tried: mod_proxy_http, mod_proxy_ajp, and mod_jk.

The problem does not occur without gzip. The failures are similar enough that I suspect the problem has to do with the hand-off from mod_deflate, though it could be further along in the request handling. It does not succeed or fail the same each request even when exactly the same content is sent with exactly the same chunk sizes, so I suspect that network timing and/or internal state may play a role.

This may be related to https://issues.apache.org/bugzilla/show_bug.cgi?id=52492 - then again, it may not. It seemed different enough to me to merit a separate issue.

This problem came from a real-world system issue that initially I suspected was an AJP problem. When transferred via mod_jk, the request body would sporadically be truncated when it was received by the backend (as confirmed by packet trace). Aside from that all appears fine.

With mod_proxy_ajp, this use case never works at all that I have seen. The entire body may be transmitted, but instead of sending the finishing 0-size chunk, httpd closes the proxy connection and returns a 200 to the client (with empty body). Errors in the log indicate mod_proxy_ajp.c(382): (20014)Internal error: ap_get_brigade failed

With mod_proxy_http, this manifests as a 0-size chunk coming in the middle of the request transfer. The backend interprets this as the end of the request (which is normally what the 0-size chunk means) and responds normally to that truncated request. Then when the proxy continues sending more chunks, the backend returns a 400 and closes the connection. The response to the client depends on timing - either the 200 response from the backend is returned, or (if the connection is closed before the request is fully sent) a 502 error is returned.

Side note: mod_proxy_http passes along the Content-Encoding: gzip header to the backend even though the traffic is not compressed at that point. This is probably a bug and might cause problems with some backends.

In order to test all this, I put together a Perl script that can gzip and chunk a request allowing me to control chunk sizes (I found that some patterns made this occur more often than others). I also created a JSP for the backend that counts the incoming bytes and returns the count in the response so that the script can verify whether the full content made it through. The content is arbitrary, but for my tests, I chose a small portion of the system dictionary. These are all attached. 

I will further attach logs and packet traces recording failures with each proxy method. These are against 2.2.21, but I have observed the same results with other versions (2.2.22, 2.2.17, 2.3.16). Again, these can be very sporadic, so when reproducing, many requests may be required before anything goes wrong (the script automates this). You may see many successes or many failures in a row. It would not surprise me if this varied by OS, libz version, architecture, etc. but I know it's not specific to my test system. In general, a good way to trigger the problem is the "randub" algorithm with a small chunk size (like 100 to send chunks with random sizes between 100 and 200).

My tests have been on Linux (CentOS 5).
$ uname -a
Linux 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:32:29 EDT 2011 i686 i686 i386 GNU/Linux
$ rpm -q zlib

Thanks for taking a look at this.
Comment 1 Luke Meyer 2012-02-03 20:07:32 UTC
Created attachment 28259 [details]
Test case and results for mod_jk method (only one success and failure from the run are included in the trace, to keep size down)
Comment 2 Luke Meyer 2012-02-03 20:08:15 UTC
Created attachment 28260 [details]
Test case and results for mod_proxy_http method
Comment 3 Luke Meyer 2012-02-03 20:08:52 UTC
Created attachment 28261 [details]
Test case and results for mod_proxy_ajp method
Comment 4 Luke Meyer 2012-02-03 21:31:12 UTC
Might help if I add that in all tests, httpd was listening at port 8000, Tomcat was listening on the same host at 8080 for http and 8009 for ajp.
Comment 5 FN 2012-08-07 17:41:13 UTC
I can confirm that mod_deflate leaves the header "Content-Encoding: gzip" in forwarded traffic when using mod_proxy. This causes BadRequest-errors when using a proxy chain in which every proxy is configured with InputFilter Deflate.

The problem of sporadic BadRequest-errors is not limited to chunked requests, we're seeing the same with normal POST-requests with gzipped body. The error rate varies.

Tested in versions 2.2.18 to 2.2.22. I'll be happy to provide additional information as needed.
Comment 6 Michael Kaufmann 2012-12-06 12:28:53 UTC
This issue is probably caused by bug 54255.
Comment 7 William A. Rowe Jr. 2018-11-07 21:09:10 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.