Bug 56541 - Proxy 502's on large file post requests
Summary: Proxy 502's on large file post requests
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.2.22
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: FixedInTrunk, MassUpdate
Depends on:
Blocks:
 
Reported: 2014-05-18 16:25 UTC by Tyler Wymer
Modified: 2018-11-07 21:10 UTC (History)
3 users (show)



Attachments
Prefetch request body before the backend connection is established/reused (trunk) (22.58 KB, patch)
2014-05-22 13:42 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (2.4.x) (22.90 KB, patch)
2014-05-22 13:44 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (trunk) (25.59 KB, patch)
2014-05-22 15:32 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (2.4.x) (25.91 KB, patch)
2014-05-22 15:39 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (trunk) (28.80 KB, patch)
2014-05-23 14:52 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (2.4.x) (29.11 KB, patch)
2014-05-23 14:53 UTC, Yann Ylavic
Details | Diff
Forward (and flush) request data as soon as available (2.2.x) (9.65 KB, patch)
2014-06-02 17:37 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (trunk) (28.27 KB, patch)
2014-06-03 15:48 UTC, Yann Ylavic
Details | Diff
Prefetch request body before the backend connection is established/reused (2.4.x) (28.68 KB, patch)
2014-06-03 15:59 UTC, Yann Ylavic
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tyler Wymer 2014-05-18 16:25:30 UTC
I'm sending post requests via the Python requests library with file attachments to our server which is using Apache mod_proxy. Originally, all posts with files over maybe 300-400kb would return a 502. I found this post: http://stackoverflow.com/questions/18854041/proxy-error-502-when-uploading-larger-files-with-ie8 and commenting out the suggested line helped, but only by seeming to raise the file size that makes it through, as now I'm seeing this with 5mb files as well. I'm making these requests from fairly slow internet (in Zambia), so I wonder if that is contributing to the timeout?

Apache config at this point:

    SSLEngine On
    SSLCertificateFile /etc/pki/tls/certs/commcarehq.org.crt
    SSLCertificateKeyFile /etc/pki/tls/private/commcarehq.org.key
    SSLCACertificateFile /etc/pki/tls/certs/hq_thawtecabundle.crt
    SSLProxyEngine On
    SetEnv proxy-sendcl 1
    RequestHeader set X-FORWARDED-PROTOCOL ssl
    RequestHeader set X-FORWARDED-SSL on
    RequestHeader set X-FORWADED-PROTO https

The post request is pretty straightforward, in case there is any potential concern from that side https://gist.github.com/twymer/69959c468f091a5471b9
Comment 1 Tyler Wymer 2014-05-19 13:58:49 UTC
Data from the logs:

[Mon May 19 13:54:19 2014] [debug] mod_proxy_balancer.c(46): proxy: BALANCER: canonicalising URL //stagingcluster/a/uth-rhd/vscan_upload
[Mon May 19 13:54:19 2014] [debug] mod_proxy_balancer.c(1014): proxy: Entering byrequests for BALANCER (balancer://stagingcluster)
[Mon May 19 13:54:19 2014] [debug] mod_proxy_balancer.c(1057): proxy: byrequests selected worker "http://10.177.30.190:9010" : busy 0 : lbstatus 0
[Mon May 19 13:54:19 2014] [debug] mod_proxy_balancer.c(581): proxy: BALANCER (balancer://stagingcluster) worker (http://10.177.30.190:9010) rewritten to http://10.177.30.190:9010/a/uth-rhd/vscan_upload
[Mon May 19 13:54:19 2014] [debug] mod_proxy.c(1020): Running scheme balancer handler (attempt 0)
[Mon May 19 13:54:19 2014] [debug] mod_proxy_http.c(1973): proxy: HTTP: serving URL http://10.177.30.190:9010/a/uth-rhd/vscan_upload
[Mon May 19 13:54:19 2014] [debug] proxy_util.c(2011): proxy: HTTP: has acquired connection for (10.177.30.190)
[Mon May 19 13:54:19 2014] [debug] proxy_util.c(2067): proxy: connecting http://10.177.30.190:9010/a/uth-rhd/vscan_upload to 10.177.30.190:9010
[Mon May 19 13:54:19 2014] [debug] proxy_util.c(2193): proxy: connected /a/uth-rhd/vscan_upload to 10.177.30.190:9010
[Mon May 19 13:54:19 2014] [debug] proxy_util.c(2444): proxy: HTTP: fam 2 socket created to connect to 10.177.30.190
[Mon May 19 13:54:19 2014] [debug] proxy_util.c(2576): proxy: HTTP: connection complete to 10.177.30.190:9010 (10.177.30.190)
[Mon May 19 13:55:20 2014] [info] [client 10.177.30.190] (32)Broken pipe: core_output_filter: writing data to the network
[Mon May 19 13:55:20 2014] [error] (103)Software caused connection abort: proxy: pass request body failed to 10.177.30.190:9010 (10.177.30.190)
[Mon May 19 13:55:20 2014] [error] proxy: pass request body failed to 10.177.30.190:9010 (10.177.30.190) from 41.223.119.17 ()
[Mon May 19 13:55:20 2014] [debug] proxy_util.c(2029): proxy: HTTP: has released connection for (10.177.30.190)
[Mon May 19 13:55:20 2014] [debug] mod_proxy_balancer.c(622): proxy_balancer_post_request for (balancer://stagingcluster)
Comment 2 Yann Ylavic 2014-05-22 13:42:18 UTC
Created attachment 31646 [details]
Prefetch request body before the backend connection is established/reused (trunk)

(In reply to Tyler Wymer from comment #1)
> [Mon May 19 13:54:19 2014] [debug] proxy_util.c(2576): proxy: HTTP:
> connection complete to 10.177.30.190:9010 (10.177.30.190)
> [Mon May 19 13:55:20 2014] [info] [client 10.177.30.190] (32)Broken pipe:
> core_output_filter: writing data to the network

It seems that your environment triggers a connection abort (by the backend) in the meantime between the connection is created/reused (by mod_proxy) and the request data are effectively sent (and flushed) on that connection.

During that meantime, mod_proxy prefetches up to 16Kbytes of the body but won't forward anything (including the request line/headers already available) until this is complete.

If that operation takes long time and/or the forwarded data are bufferized in httpd's output queue until some size, the backend may timeout the connection and close it.

Could you try the attached patch which moves the prefetch before the backend connection is established/reused (so to minimize the race condition between establishing/reusing a connection and sending data on it), and allows to SetEnv proxy-flushall to tell mod_proxy not to flush everything forwarded?
Comment 3 Yann Ylavic 2014-05-22 13:44:57 UTC
Created attachment 31647 [details]
Prefetch request body before the backend connection is established/reused (2.4.x)

Same patch for 2.4.x.
Comment 4 Yann Ylavic 2014-05-22 14:09:03 UTC
Please note that for this patch to work, the request should not be spooled on the disk before being forwarded.

That may arise if you use "SetEnv proxy-sendcl" and the request is "Transfer-Encoding: chunked" for example, or if the length can't be determined when mod_proxy has to forward the HTTP header (INFLATE filter in the chain, ...).

Why do you use proxy-sendcl first?
Comment 5 Yann Ylavic 2014-05-22 15:32:42 UTC
Created attachment 31648 [details]
Prefetch request body before the backend connection is established/reused (trunk)

Finally this new patch takes also care of the spooled requests, by spooling them (completely) during prefetch (before the connect/reuse-then-forward window).
Hence SetEnv proxy-sendcl shouldn't matter.
Comment 6 Yann Ylavic 2014-05-22 15:39:40 UTC
Created attachment 31649 [details]
Prefetch request body before the backend connection is established/reused (2.4.x)

Same for 2.4.x.
Comment 7 Yann Ylavic 2014-05-23 14:52:14 UTC
Created attachment 31654 [details]
Prefetch request body before the backend connection is established/reused (trunk)

This version fixes input brigades lifetime issues.
Comment 8 Yann Ylavic 2014-05-23 14:53:14 UTC
Created attachment 31655 [details]
Prefetch request body before the backend connection is established/reused (2.4.x)

Same for 2.4.x.
Comment 9 Tyler Wymer 2014-05-29 16:12:24 UTC
Yann,

Thanks for the response. We are actually running Apache 2.2.22 (I neglected to specify that on the bug but updated it now). Is it possible to get a version of this patch that would work for 2.2.x?
Comment 10 Yann Ylavic 2014-06-02 17:37:55 UTC
Created attachment 31684 [details]
Forward (and flush) request data as soon as available (2.2.x)

The corresponding (complete) patch against 2.2.x would probably not be accepted because of the changes on mod_proxy_http which may affect existing configurations (although it should not, so much configurations work without it until now).

Hence, a lightweight version of the patch is attached for 2.2.x, which will simply disable any buffering of the request before it is forwarded to the backend (so that there is a minimal delay between the backend connection is established/reused and the request is sent on it).

This should prevent the backend from closing the connection because of a timeout (the time taken by mod_proxy to pre-fetch the request).
You will have to "SetEnv proxy-flushall" to enable it.

Please note that it won't work if you SetInputFilter something, since in this case the Content-Length might change and mod_proxy won't forward it as is (the request will be spooled on disk).

The same applies regarding "SetEnv proxy-sendcl", it is usually needed because one wants "Transfer-Encoding: chunked" from the client to be turned into Content-Length (eg. the backend does not support chunked requests), but in this case the request will also be spooled, and the patch is useless too.
So either you can #comment it, or you need the full patch (2.4.x).

Finally, there is still race condition where the backend may close a reused connection at the same time when mod_proxy checks/thinks it is still alive.
To avoid that, you have to configure the worker's TTL (ProxyPass ... ttl=<seconds>) with a lower value than the KeepAliveTimeout used by the backend, so that no connection is reused if it is idle for more than TTL seconds.
Comment 11 Yann Ylavic 2014-06-02 17:56:58 UTC
(In reply to Yann Ylavic from comment #10)
> To avoid that, you have to configure the worker's TTL (ProxyPass ...
> ttl=<seconds>) with a lower value than the KeepAliveTimeout used by the
> backend, so that no connection is reused if it is idle for more than TTL
> seconds.

More precisely, the ProxyPass should be :
   ProxyPass ... ttl=<seconds> smax=0
Comment 12 Ruediger Pluem 2014-06-02 18:55:06 UTC
(In reply to Yann Ylavic from comment #11)

> More precisely, the ProxyPass should be :
>    ProxyPass ... ttl=<seconds> smax=0

Why smax=0?
Comment 13 Yann Ylavic 2014-06-02 20:17:07 UTC
(In reply to Ruediger Pluem from comment #12)
> (In reply to Yann Ylavic from comment #11)
> 
> > More precisely, the ProxyPass should be :
> >    ProxyPass ... ttl=<seconds> smax=0
> 
> Why smax=0?

Well, it seems to me that apr_reslist won't apply ttl unless nidle > smax, but mod_proxy sets undefined smax to hmax, hence hmax > nidle is always false (and ttl is never applied by default). With smax=0, all idle connections are considered.
Comment 14 Yann Ylavic 2014-06-02 20:20:52 UTC
(In reply to Yann Ylavic from comment #13)
> hence hmax > nidle is always false
I meant nidle > hmax here.
Comment 15 Yann Ylavic 2014-06-02 20:39:52 UTC
(In reply to Yann Ylavic from comment #13)
> > Why smax=0?
> 
> Well, it seems to me that apr_reslist won't apply ttl unless nidle > smax,
> but mod_proxy sets undefined smax to hmax, hence hmax > nidle is always
> false (and ttl is never applied by default). With smax=0, all idle
> connections are considered.

Hm, after verification (should have done that before posting), this is true for apr_reslist_release (in fact apr_reslist_maintain), but not for apr_reslist_acquire. So it works without smax=0 too...
Comment 16 Yann Ylavic 2014-06-03 15:48:43 UTC
Created attachment 31686 [details]
Prefetch request body before the backend connection is established/reused (trunk)

Fixes the chunk_hdr (transient) scope in stream_reqbody_chunked() and don't let proxy-flushall interfere with spooled-CL vs forwarded-CL/TE decision.
Comment 17 Yann Ylavic 2014-06-03 15:59:33 UTC
Created attachment 31687 [details]
Prefetch request body before the backend connection is established/reused (2.4.x)

Likewise (2.4.x).
Comment 18 Andre W. 2015-01-28 15:23:12 UTC
Is there a possibility, that this functinallity will be part of one of the next Apache 2.4.x releases? Because of "mod_proxy does not flush data on POST requests from client" (https://issues.apache.org/bugzilla/show_bug.cgi?id=37920)

Cheers,
André
Comment 19 Yann Ylavic 2015-02-01 00:50:04 UTC
Committed in r1656259 (trunk).
Comment 20 Paul Nyheim 2017-03-02 12:24:11 UTC
I tried the 2.4.x patch, but it does not apply cleanly against 2.4.25 which we are running on.

1. Is it possible to get a patch that applies to 2.4.25?
2. Is this issue considered for an upcoming 2.4.x release?

Regards,
Paul Nyheim
Comment 21 William A. Rowe Jr. 2018-11-07 21:10:00 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.