With an apahce proxying requests over ajp to a backend tomcat, the following sequence of events is possible: User1 requests /badger, and sends headers. Request hits apache, filters into ap_proxy_ajp_request in mod_proxy_ajp.c User1 terminates connection with extreme prejudice. Apache sends headers to tomcat. Tomcat takes it's time. Apache tries to read the rest of the request body with ap_get_brigade, mod_proxy_ajp.c:174. This fails as the underlying socket cannot recv any more. So ap_proxy_ajp_request returns 500 immediatly. conn->close++ is not called so the connection is not touched, merely put back in the available connections. Apache now fails int he output filters trying to send the status 500 error message. Now user2 connects, requests /moose, apache takes the above connection from the pool and sends the headers down the pipe to the tomcat. The tomcat now returns /badger, as that's finally finished processing. Apache in ajp_read_header reads the SEND_BODY_CHUNK message, and processes it. Then repeats reading messages until the END_RESPONSE message. Whereupon apache has successfully served /badger to someone who requested / moose. Needless to say this is a critical flaw with the ajp connection handling.
Created attachment 18750 [details] Patch that attempts to fix the bug This patch _should_, if my understanding of the bug is correct , fix it.
Well spotted! Committed to trunk as r434483 (http://svn.apache.org/viewvc?view=rev&revision=434483). Thanks for the patch.
Proposed for backport to 2.2.x as r434488 http://svn.apache.org/viewvc?view=rev&revision=434488).
I'd like to add that in our shop we witnessed this exact same problem with Apache 2.2.3 and mod_jk 1.2.19, not mod_proxy_ajp. Will the fix being applied cure this problem when using mod_jk or is a seperate patch going to be applied to that module as well? Thanks.
No this will not cure mod_jk. Bugs for mod_jk should not be further handled in this report. Please open a new report with Product: Tomcat 5 and Component: Native:JK. BTW: The fix for the original bug will be part of 2.2.4 as it has been backported.
The problem is still not fixed correctly in 2.2.4 (the worker is marked errored and connection retried but that is wrong): +++ Fri Mar 16 08:27:06 2007] [error] [client 71.140.198.6] proxy: error processing body, referer: https://xxx.yyy.zzz/site/checkout/ship_method.html [Fri Mar 16 08:27:06 2007] [error] proxy: got bad response (5) from 64.85.80.16:8009 (app4) [Fri Mar 16 08:27:06 2007] [error] proxy: BALANCER: (balancer://appservers). All workers are in error state for route (app4-engine1) +++ I have fixed it in trunk.
Created attachment 19772 [details] patch for 2.2.x I will commit it in the 2.2.x branch if noone complains.
Fix backported to 2.2.x as r553593 (http://svn.apache.org/viewvc?view=rev&revision=553593)
Here's a report for the same symptom with httpd-2.2.11: http://www.tomcatexpert.com/blog/2010/06/16/deciding-between-modjk-modproxyhttp-and-modproxyajp#comment-207 Meanwhile, I just experienced the same with httpd-2.4.2 I can't rule out the possibility of bug in my application code, misbehaving proxy server, etc. However, the mixed up response only happens with mod_proxy_ajp, not mod_proxy_http.
I have logged a separate issue: bug 53727