|Summary:||mod_proxy_http fails with 502 when backend sends 401 and closes connection immediately|
|Product:||Apache httpd-2||Reporter:||Bruno Harbulot <bruno>|
|Component:||mod_proxy_http||Assignee:||Apache HTTPD Bugs Mailing List <bugs>|
|Severity:||normal||CC:||apache, michaelo, szg0000|
Description Bruno Harbulot 2017-02-09 12:32:56 UTC
There seems to be two main aspects to this problem: 1. How mod_proxy_http handles a backend sending a TCP RST. 2. How mod_proxy_http handles a 401 response code from the backend, especially in relation to a "Expect: 100-continue" in the request. The test case is an Apache Httpd server (2.4.25) used as a front-end to a Jetty (9.3.16) server, using mod_proxy_http as a reverse proxy. The mod_proxy configuration is as follows: SetEnv HTTPS 1 <Location /test/> ProxyPass http://localhost:8080/test/ retry=10 ProxyPassReverse http://localhost:8080/test/ RequestHeader set X-Forwarded-Proto "https" env=HTTPS RequestHeader set X-Forwarded-Port 443 env=HTTPS </Location> The Jetty service is configured to use HTTP Basic authentication. When using Curl to send an external POST request using a wrong username/password, I get a 502 status code from Apache Httpd. Here is what is seen from the external client: > POST /test/ HTTP/1.1 > Authorization: Basic ... > User-Agent: curl/... > Host: test.example.com > Accept: application/xml > Referer: https://test.example.com/test/ > Content-Type: application/xml > Content-Length: 40220 > Expect: 100-continue > < HTTP/1.1 100 Continue < HTTP/1.1 502 Bad Gateway < Date: Thu, 09 Feb 2017 11:27:03 GMT < Server: Apache/2.4.25 < Content-Length: 232 < Content-Type: text/html; charset=iso-8859-1 * HTTP error before end of send, stop sending Here is what was sent between Apache Httpd and the Jetty server locally: A: POST /test/ HTTP/1.1 A: Host: test.example.com A: Authorization: Basic .... A: User-Agent: curl/... A: Accept: application/xml A: Referer: https://test.example.com/test/ A: Content-Type: application/xml A: Expect: 100-continue A: X-Forwarded-Proto: https A: X-Forwarded-Port: 443 A: X-Forwarded-For: .... A: X-Forwarded-Host: .... A: X-Forwarded-Server: ... A: Connection: Keep-Alive A: Content-Length: 40220 A: J: HTTP/1.1 401 Bad credentials J: X-Content-Type-Options: nosniff J: X-XSS-Protection: 1; mode=block J: Pragma: no-cache J: Strict-Transport-Security: max-age=31536000 ; includeSubDomains J: X-Frame-Options: SAMEORIGIN J: WWW-Authenticate: Basic realm="Realm" J: Cache-Control: must-revalidate,no-cache,no-store J: Content-Length: 0 J: Connection: close J: A: <?xml version="1.0" encoding="UTF-8"?> A: ... Here is what the Wireshark packet summary looks like: No. Time Destination Port Protocol Length Info 1 0.000000 8080 TCP 94 43646 ? 8080 [SYN] Seq=0 Win=65476 Len=0 MSS=65476 SACK_PERM=1 TSval=577106868 TSecr=0 WS=128 2 0.000084 43646 TCP 94 8080 ? 43646 [SYN, ACK] Seq=0 Ack=1 Win=65464 Len=0 MSS=65476 SACK_PERM=1 TSval=577106868 TSecr=577106868 WS=128 3 0.000168 8080 TCP 86 43646 ? 8080 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=577106868 TSecr=577106868 4 0.036775 8080 HTTP 642 POST /test/ HTTP/1.1 5 0.036836 43646 TCP 86 8080 ? 43646 [ACK] Seq=1 Ack=557 Win=66688 Len=0 TSval=577106905 TSecr=577106905 6 0.039358 43646 HTTP 423 HTTP/1.1 401 Bad credentials 7 0.039562 43646 TCP 86 8080 ? 43646 [FIN, ACK] Seq=338 Ack=557 Win=66688 Len=0 TSval=577106908 TSecr=577106905 8 0.043988 8080 TCP 86 43646 ? 8080 [ACK] Seq=557 Ack=338 Win=66560 Len=0 TSval=577106912 TSecr=577106907 9 0.046297 8080 TCP 24662 [TCP segment of a reassembled PDU] 10 0.046390 43646 TCP 74 8080 ? 43646 [RST] Seq=339 Win=0 Len=0 11 0.047071 8080 TCP 86 43646 ? 8080 [RST, ACK] Seq=25133 Ack=339 Win=66560 Len=0 TSval=577106915 TSecr=577106908 Part of the problem here is that Jetty almost immediately sends a TCP RST, after sending its 401 response (with a "Connection: close"), partly to prevent DOS attacks due to unauthenticated requests: https://github.com/eclipse/jetty.project/issues/651 Where I think there may be a problem on Apache Httpd's side. The original client sent an "Expect: 100-continue" header, and that was forwarded by mod_proxy. Yet, Jetty replied with a 401 response before getting any of the request's entity, and certainly before sending a 100 response. I believe in this case that mod_proxy should: - not send "HTTP/1.1 100 Continue" back to the client - not carry on with sending the request's entity to Jetty On top of this, because Jetty sends a TCP RST, this causes mod_proxy to send a 502 back to the client, instead of the 401 (with headers) it already received from the backend. A consequence is that some clients that don't use pre-emptive HTTP Basic authentication (i.e. those that will only send the Authorization header when challenged with a 401 response) will just take this 502 response as a failure, instead of trying again with credentials. There already are a couple of issues related to this: - bug 51867 concluded (rightly, I think) that the backend "needs to consume the body of requests even if it does not need them". However, in this case we're using "Expect: 100-continue" and the 401 response is sent before the request entity is sent. - bug 49405: although sending a TCP RST is indeed a bit abrupt, this is done after sending a full valid 401 response (with all headers, "Content-Length: 0", and "Connection: close), before any request entity was sent to the backend. It should at least return that instead of 502 (especially w.r.t. 100-continue issue).
Comment 1 Michael Osipov 2019-08-02 09:29:14 UTC
I think this is fixed by BZ 60330.
Comment 2 Michael Osipov 2020-05-19 14:06:32 UTC
I must revert my comment. It is not fixed by the noted issue.
Comment 3 Michael Osipov 2020-05-20 19:30:01 UTC
Here is another in-detail description of the issue: https://firstname.lastname@example.org/msg135207.html
Comment 4 Yann Ylavic 2020-05-20 20:30:37 UTC
This is fixed in 2.4.43 with end to end 100-continue negotiation. If the backend (Jetty) responds with a final status to a 100-continue request then mod_proxy will forward the response without ever trying to forward the request body.