Bug 60330 - reverse proxy of 100-continue aware backend, sends 100 continue even when backend does not
Summary: reverse proxy of 100-continue aware backend, sends 100 continue even when bac...
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy_http (show other bugs)
Version: 2.4.23
Hardware: PC All
: P2 normal with 6 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: FixedInTrunk
: 55433 57853 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-11-02 16:07 UTC by Jay R. Wren
Modified: 2018-09-27 21:03 UTC (History)
5 users (show)



Attachments
in the reverse proxy case, if request had Expects: 100-continue, delay writing 100 continue response until backend has sent 100-continue response (4.87 KB, patch)
2016-11-11 22:28 UTC, Jay R. Wren
Details | Diff
Forward 100-continue (and minimize race when reusing backend connections) (47.09 KB, patch)
2016-11-16 00:07 UTC, Yann Ylavic
Details | Diff
curl(1) to Tomcat directly (3.93 KB, text/plain)
2018-07-06 10:57 UTC, Michael Osipov
Details
curl(1) to Tomcat via HTTPd (2.95 KB, text/plain)
2018-07-06 10:57 UTC, Michael Osipov
Details
web.xml (to put in webapps/ROOT/WEB-INF/) (1.78 KB, application/xml)
2018-07-23 20:28 UTC, jfclere
Details
jsp to demonstrate the issue (put it in webapps/ROOT) (418 bytes, text/plain)
2018-07-23 20:29 UTC, jfclere
Details
Patch for trunkthat works for me (based on Yann one) (44.88 KB, patch)
2018-07-24 07:32 UTC, jfclere
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jay R. Wren 2016-11-02 16:07:57 UTC
Reverse proxy of 100-continue aware backend, sends 100 continue even when backend does not. This causes a client to think it should write a request body, while the backend may still respond with a 400 and not read the request body. mod_proxy_http then responds with 502 as a result of AH01097: pass request body failed

The backend is doing the right thing: it did not send a 100 continue so it should not be required to read a request body, regardless of transfer encoding or content-length.

Expected:

mod_proxy_http reverse proxy should not send 100-continue to a client unless the backend does.
Comment 1 Jay R. Wren 2016-11-11 22:28:03 UTC
Created attachment 34438 [details]
in the reverse proxy case, if request had Expects: 100-continue, delay writing 100 continue response until backend has sent 100-continue response
Comment 2 Yann Ylavic 2016-11-16 00:07:40 UTC
Created attachment 34451 [details]
Forward 100-continue (and minimize race when reusing backend connections)

I proposed this patch a while ago on the dev@ list ([1]), this is an update for latest trunk, with more (though incomplete) testing.

Could you please give it a try?

[1]. https://lists.apache.org/thread.html/4e541e032b8a77ebec8248534637b47cdcd4f38af79baa5259845db0@1430360070@%3Cdev.httpd.apache.org%3E
Comment 3 Jim Jagielski 2016-11-21 20:25:22 UTC
*** Bug 55433 has been marked as a duplicate of this bug. ***
Comment 4 Jay R. Wren 2016-12-09 03:08:26 UTC
Yann,

I tried that patch, but I still get 503 error when expecting a 100 Continue response.
Comment 5 Michael Osipov 2018-07-04 16:04:20 UTC
Any chance that this will be fixed? Have the very same problem from a backend Tomcat. I guess I need to drop mod_proxy and try mod_ajp or drop Apache HTTPd altogether for this.
Comment 6 Yann Ylavic 2018-07-04 21:05:01 UTC
*** Bug 57853 has been marked as a duplicate of this bug. ***
Comment 7 Yann Ylavic 2018-07-04 21:18:54 UTC
Sorry it's been a long time, I think we need more informations here as to the exact issue.

What exactly isn't working with the proposed patch?
Where are 100-continue or request bodies lacking or sent inappropriately, on which side (client/backend)?
IOW, can we please have a description/scenario of what is supposed to work and how, possibly with the expected request/response on both sides?
What is the configuration being tested?
Comment 8 Michael Osipov 2018-07-06 08:05:17 UTC
(In reply to Yann Ylavic from comment #7)
> Sorry it's been a long time, I think we need more informations here as to
> the exact issue.
> 
> What exactly isn't working with the proposed patch?
> Where are 100-continue or request bodies lacking or sent inappropriately, on
> which side (client/backend)?
> IOW, can we please have a description/scenario of what is supposed to work
> and how, possibly with the expected request/response on both sides?
> What is the configuration being tested?

Hi Yann,

I can provide a full verbose log of curl(1) for Tomcat behind HTTPd with faulty behavior and direct Tomcat communication. Moreover, I can expore the httpd.conf for that offending behavior.
Comment 9 Yann Ylavic 2018-07-06 08:28:23 UTC
Yes please do, along with the httpd error_log with LogLevel trace7.
Thanks!
Comment 10 Michael Osipov 2018-07-06 10:57:04 UTC
Created attachment 36015 [details]
curl(1) to Tomcat directly
Comment 11 Michael Osipov 2018-07-06 10:57:28 UTC
Created attachment 36016 [details]
curl(1) to Tomcat via HTTPd
Comment 12 Michael Osipov 2018-07-06 11:01:12 UTC
The error log has been sent privately due to sensitive data. Looking forward to an analysis.
Comment 13 Michael Osipov 2018-07-06 11:43:39 UTC
FYI, I have tried mod_proxy_{http,ajp} and mod_jk.
Comment 14 Yann Ylavic 2018-07-06 11:56:55 UTC
Thanks Michael, at first glance the error_log is with mod_proxy_ajp, while attachment 34451 [details] is about mod_proxy_http (and I'd like to keep the scope there for now).
I agree that unpatched mod_proxy_http sends "100 continue" too soon (actually independently on the client and backend side).
The patch is precisely to avoid that (hop by hop 100-continue handling), did you give it a try? If yes, could I have the error_log with mod_proxy_http?
Comment 15 Michael Osipov 2018-07-06 19:19:47 UTC
(In reply to Yann Ylavic from comment #14)
> Thanks Michael, at first glance the error_log is with mod_proxy_ajp, while
> attachment 34451 [details] is about mod_proxy_http (and I'd like to keep the
> scope there for now).
> I agree that unpatched mod_proxy_http sends "100 continue" too soon
> (actually independently on the client and backend side).
> The patch is precisely to avoid that (hop by hop 100-continue handling), did
> you give it a try? If yes, could I have the error_log with mod_proxy_http?

I agree, I have tried all possible modules with the same negative result. I will redo for you. Moreover, I will compile from trunk along with your patch and try to reproduce. Does it still apply cleanly to trunk?

Please be patient, I won't be able to test anything before 2018-07-23.

Thank you.
Comment 16 jfclere 2018-07-23 20:28:09 UTC
Created attachment 36046 [details]
web.xml (to put in webapps/ROOT/WEB-INF/)
Comment 17 jfclere 2018-07-23 20:29:24 UTC
Created attachment 36047 [details]
jsp to demonstrate the issue (put it in webapps/ROOT)
Comment 18 jfclere 2018-07-23 20:39:12 UTC
when using the web.xml and red.jsp and having
<tomcat-users xmlns="http://tomcat.apache.org/xml"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
              version="1.0">
  <role rolename="manager"/>
  <user username="tomcat" password="tomcat" roles="manager"/>
</tomcat-users>
in conf/tomcat-users.xml in tomcat you can get the problem with basic authentication:
curl --verbose --anyauth -u tomcat:tomcat http://localhost:8000/read.jsp -X POST -T file -H "Content-Type: text/plain"

file needs to be big enough ~8.5M

you will get:
[jfclere@dhcp-144-173 build]$ curl --verbose --anyauth -u tomcat:tomcat http://localhost:8000/read.jsp -X POST -T toto -H "Content-Type: text/plain"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
> POST /read.jsp HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Type: text/plain
> Content-Length: 8849904
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 504 Gateway Timeout
< Date: Mon, 23 Jul 2018 20:38:21 GMT
< Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.0h-fips
< Content-Length: 247
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>504 Gateway Timeout</title>
</head><body>
<h1>Gateway Timeout</h1>
<p>The gateway did not receive a timely response
from the upstream server or application.</p>
</body></html>
* Connection #0 to host localhost left intact
Comment 19 jfclere 2018-07-24 07:32:13 UTC
Created attachment 36048 [details]
Patch for trunkthat works for me (based on Yann one)
Comment 20 Yann Ylavic 2018-07-24 21:58:12 UTC
Thanks Jean-Frédéric, applied in r1836588.

This is the same patch (plus a req != NULL check) rebased on latest trunk, right?
Comment 21 Michael Osipov 2018-07-25 12:44:22 UTC
Commit on trunk does not work for me. I will send logs privately to Yann.
Comment 22 Yann Ylavic 2018-07-25 16:41:23 UTC
Thanks for testing Michael.

Fixed in r1836648 (hopefully).
Comment 23 jfclere 2018-08-06 12:12:15 UTC
Fixed according to my tests.
Comment 24 Michael Osipov 2018-09-27 21:03:41 UTC
Can we please backport this to 2.4.x branch? It is not present in 2.4.35 :-(