Bug 62362 - Proxied content not properly rate-limited
Summary: Proxied content not properly rate-limited
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_ratelimit (show other bugs)
Version: 2.4.33
Hardware: PC Mac OS X 10.1
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Keywords: FixedInTrunk
Depends on:
Reported: 2018-05-08 20:41 UTC by Luca Toscano
Modified: 2018-08-30 07:40 UTC (History)
0 users

httpd-trunk-mod_ratelimit-rate_limit_proxied_content.patch (6.25 KB, patch)
2018-06-03 06:43 UTC, Luca Toscano
Details | Diff
httpd-trunk-mod_ratelimit-rate_limit_proxied_content.patch (7.30 KB, patch)
2018-06-06 16:31 UTC, Luca Toscano
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Luca Toscano 2018-05-08 20:41:26 UTC
On users@ it was reported the following scenario:

I'm using Apache 2.4.24 on Debian 9 Stable, behind a DSL connection, with an estimated upload capacity of ~130kB/s.
I'm trying to limit the bandwidth available to my users (per-connection limit is fine).
However, it seems to me that the rate-limit parameter is coarsely grained :

- if I set it to 8, users are limited to 8 kB/s
- if I set it to 20, or 30, users are limited to 40 kB/s
- if I set it to 50, 60 or 80, users are limited to my BW, so ~120 kB/s

After following up with the user it seems that the issue happens with proxied content. So I've set up the following experiment:

- Directory with a 4MB file inside
- Simple Location that proxies content via mod_proxy_http to a Python process running a webserver, capable of returning the same 4MB file outlined above.

I tested the rate limit using curl's summary (average Dload speed for example).

This is what I gathered:

- when httpd serves the file directly, mod_ratelimit's output filter is called once and the bucket brigade contains all the data contained in the file. This is probably due to how bucket brigates work when morphing a file content?

- when httpd serves the file via mod_proxy, the output filter is called multiple times, and each time the buckets are maximum the size of ProxyIOBufferSize (8192 by default). Still not completely sure about this one, so please let me know if I am totally wrong :)

The main problem is, IIUC, in the output's filter logic that does this: it calculates the size of a chunk, based on the rate-limit set in the httpd's conf, and then it splits the bucket brigade, if necessary, in buckets of that chunk size, interleaving them with FLUSH buckets (and sleeping 200ms).

So a trace of execution with say a chunk size of 8192 would be something like:

First call of the filter: 8192 --> FLUSH --> sleep(200ms) --> 8192 --> ... -> last chunk (either 8192 or something less).

This happens correctly when httpd serves directly the content, but not when proxied:

First call of the filter: 8192 -> FLUSH (no sleep, since do_sleep turns to 1 only after the first flush)

Second call of the filter: 8192 -> FLUSH (no sleep)


So one way to alleviate this issue is to move do_sleep to the ctx data structure, so if the filter gets called multiple times it will "remember" to sleep between flushes (with the assumption that it is allocated for each request). It remains the problem that when the rate-limit speed sets a chunk size different than the ProxyIOBufferSize (8192 by default) then the client will be rate limited to the speed dictated by the buffer size (for example, 8192 should correspond to ~40KB/s).
Comment 1 Luca Toscano 2018-06-03 06:43:32 UTC
Created attachment 35950 [details]
Comment 2 Luca Toscano 2018-06-03 07:09:14 UTC
The attached patch seems to work on my test case scenarios. What I did was:

* created a ~4M text file containing lines like TEST 0, TEST 1, TEST 2, etc..
* served the file directly from httpd (via Directory config) and also via Python 3's http server and proxied via mod_proxy_http.
* calculated the sha1sum of the files generated redirecting curl's output to a file.
* noted time spent in both cases to rate limit the content with various rate limit speeds (via curl's average dload speed).

Everything seems consistent and fine as far as I can see, the patch seems to work as expected. This is a rather invasive refactoring of the rate-limit filter so I might have missed something, testing/feedback/suggestions/reviews are really welcome :)
Comment 3 Luca Toscano 2018-06-06 16:31:43 UTC
Created attachment 35957 [details]
Comment 4 Luca Toscano 2018-06-19 22:30:28 UTC
Committed r1833875 in httpd trunk, feedback/testing is welcome!
Comment 5 Luca Toscano 2018-07-07 07:25:40 UTC
Backported to 2.4.x in http://svn.apache.org/r1835168, the fix will be part of 2.4.34.
Comment 6 Luca Toscano 2018-08-30 07:40:52 UTC
For posterity: this change lead to https://bz.apache.org/bugzilla/show_bug.cgi?id=62568