Overview -------- The mod_reqtimeout module is not dropping connections and returning 408 when dealing with "slow http header" or "slow http body" requests. Instead, it is either truncating the request and handling it, or dropping the request with a 400 status code. Steps to reproduce (A) ---------------------- 1. Launch a slow-post attack using the OWASP HTTP DoS tool (http://code.google.com/p/owasp-dos-http-post/downloads/list) http_dos_cli --host 1.2.3.4 --port 80 --path /server-status --slow-post --post-field j_username --connections 1000 --rate 1000 --timeout 5 2. Sniff network traffic using Wireshark, observe requests being truncated and handled, resulting in a 200 return code. Steps to reproduce (B) ---------------------- 1. Launch a slow-headers attack 2. Sniff network trafic using Wireshark, observe requests being dropped with a 400 code being returned. Expected behavior ----------------- Request is dropped and a 408 status code is returned. Actual behavior --------------- Request is dropped and a 400 status code is returned OR request is truncated and handled normally. Platform -------- Software: Apache 2.2.17 (MPM-worker) OS: Solaris 5.10 32-bit Hardware: Sun SPARC Additional information ---------------------- mod_reqtimeout configuration RequestReadTimeout header=10-20,MinRate=500 body=10-20,MinRate=500 ModSecurity 2.5.13 is also configured. Apache debug logs show that incoming requests time out: [Tue Apr 19 08:55:09 2011] [info] [client 5.6.7.8] Request header read timeout [Tue Apr 19 08:55:09 2011] [error] [client 5.6.7.8] request failed: error reading the headers OR [Tue Apr 19 09:01:20 2011] [info] [client 5.6.7.8] Request body read timeout --- Thanks!
(In reply to comment #0) > Steps to reproduce (A) > ---------------------- > 1. Launch a slow-post attack using the OWASP HTTP DoS tool > (http://code.google.com/p/owasp-dos-http-post/downloads/list) > http_dos_cli --host 1.2.3.4 --port 80 --path /server-status --slow-post > --post-field j_username --connections 1000 --rate 1000 --timeout 5 > 2. Sniff network traffic using Wireshark, observe requests being truncated and > handled, resulting in a 200 return code. I couldn't reproduce this (but I don't have windows to actually try the tool). Can you provide the wireshark dump (maybe filtered to only contain one request)? Do you have mod_status listening for /server-status? > Steps to reproduce (B) > ---------------------- > 1. Launch a slow-headers attack > 2. Sniff network trafic using Wireshark, observe requests being dropped with a > 400 code being returned. This happens in various situations and is fixed in trunk. The fixes should probably be backported to 2.2.x. The relevant commits are r820760 r919323 r937858 r938265
Created attachment 26927 [details] Slow post Wireshark dump
Hi Stefan and thanks for taking the time to look at this. (In reply to comment #1) > (In reply to comment #0) > > Steps to reproduce (A) > > ---------------------- > > 1. Launch a slow-post attack using the OWASP HTTP DoS tool > > (http://code.google.com/p/owasp-dos-http-post/downloads/list) > > http_dos_cli --host 1.2.3.4 --port 80 --path /server-status --slow-post > > --post-field j_username --connections 1000 --rate 1000 --timeout 5 > > 2. Sniff network traffic using Wireshark, observe requests being truncated and > > handled, resulting in a 200 return code. > > I couldn't reproduce this (but I don't have windows to actually try the tool). > Can you provide the wireshark dump (maybe filtered to only contain one > request)? Do you have mod_status listening for /server-status? I have attached a Wireshark dump to the bug report. Let me know if this is what you expected, I'm actually new to Wireshark. We have mod_status listening on /server-status and it's responding correctly when invoking with a browser. Apache is now returning a 400 code, similar to the slow-headers attack. We did tweak a few settings (disabled ModSecurity, turned off ExtendedStatus) so it might have had that effect. I will investigate further. > > Steps to reproduce (B) > > ---------------------- > > 1. Launch a slow-headers attack > > 2. Sniff network trafic using Wireshark, observe requests being dropped with a > > 400 code being returned. > > This happens in various situations and is fixed in trunk. The fixes should > probably be backported to 2.2.x. The relevant commits are r820760 r919323 > r937858 r938265 This is good to know. Do you have any idea when these changes will be backported or when 2.3 will be released?
(In reply to comment #3) > I have attached a Wireshark dump to the bug report. Let me know if this is > what you expected, I'm actually new to Wireshark. Thanks, that was what I wanted. > We have mod_status listening on /server-status and it's responding correctly > when invoking with a browser. > > Apache is now returning a 400 code, similar to the slow-headers attack. We did > tweak a few settings (disabled ModSecurity, turned off ExtendedStatus) so it > might have had that effect. I will investigate further. But as you say, it does not show a 200 result code. I will mark this as invalid because for the other issues, there have already been different PRs. If you can reproduce the code 200 sent despite of a timeout issue, please change the status from INVALID to REOPENED. > > This happens in various situations and is fixed in trunk. The fixes should > > probably be backported to 2.2.x. The relevant commits are r820760 r919323 > > r937858 r938265 > > This is good to know. Do you have any idea when these changes will be > backported or when 2.3 will be released? I have now proposed them for the next 2.2.x release. 2.4 is hopefully due in 1-2 months.
Created attachment 26935 [details] Slow post Wireshark dump with 302 code
(In reply to comment #4) > (In reply to comment #3) > But as you say, it does not show a 200 result code. I will mark this as invalid > because for the other issues, there have already been different PRs. If you can > reproduce the code 200 sent despite of a timeout issue, please change the > status from INVALID to REOPENED. Ok, I have reproduced the problem, although it's slightly different from what I initially wrote. You will see in the new attachment that the request times out but returns a 302 Redirect instead of a 400 Bad Request. A valid request should indeed return a 302 because of a RedirectMatch rule in the httpd-vhosts.conf file, but the request times out and a 302 is returned anyway. Also, you will see that the client keeps sending data after the connection is closed - although this may be an issue with the http_dos_cli tool.
(In reply to comment #6) > You will see in the new attachment that the request times out but returns a 302 > Redirect instead of a 400 Bad Request. A valid request should indeed return a > 302 because of a RedirectMatch rule in the httpd-vhosts.conf file, but the > request times out and a 302 is returned anyway. Also, you will see that the > client keeps sending data after the connection is closed - although this may be > an issue with the http_dos_cli tool. And because of this, the http_dos_cli tool can continue sending data slowly and hogging worker threads. The ModSecurity rule that checks for 408 bursts from a single IP cannot know that this is coming from a slow http DoS attack, and so cannot drop further connections from that IP.
Thanks for the additional dump. This is definitely a bug in httpd, but I am not sure yet how to fix it. The problem is that ap_finalize_request_protocol() ignores errors returned by ap_discard_request_body(). But simply calling ap_die() inside ap_finalize_request_protocol() causes various test failures.
With r1103213 in trunk, the connection will be closed correctly after a timeout during a request body that is discarded. It still means that the request is logged as successful an not as 408, though. One could argue that this is correct because the request was completed successfully. But I see that other modules may be interested if there was a timeout. I will have to think some more about that.
r1103213 has been backported to 2.2.20 as r1162862.
(In reply to Stefan Fritsch from comment #9) > ...I will have to think some more about that. ... but this is now 7 years ago :) This is part of 2.2.20 and 2.4.x from the beginning. So closing as fixed.