Bug 60330

Summary: reverse proxy of 100-continue aware backend, sends 100 continue even when backend does not
Product: Apache httpd-2 Reporter: Jay R. Wren <jrwren>
Component: mod_proxy_httpAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: critical CC: jim, jon.abourbih, michaelo, post, s.buesing, shm-dmitry
Priority: P2 Keywords: FixedInTrunk
Version: 2.4.23   
Target Milestone: ---   
Hardware: PC   
OS: All   
Attachments: in the reverse proxy case, if request had Expects: 100-continue, delay writing 100 continue response until backend has sent 100-continue response
Forward 100-continue (and minimize race when reusing backend connections)
curl(1) to Tomcat directly
curl(1) to Tomcat via HTTPd
web.xml (to put in webapps/ROOT/WEB-INF/)
jsp to demonstrate the issue (put it in webapps/ROOT)
Patch for trunkthat works for me (based on Yann one)
apache-trace7-dumpio logfile

Description Jay R. Wren 2016-11-02 16:07:57 UTC 
Reverse proxy of 100-continue aware backend, sends 100 continue even when backend does not. This causes a client to think it should write a request body, while the backend may still respond with a 400 and not read the request body. mod_proxy_http then responds with 502 as a result of AH01097: pass request body failed

The backend is doing the right thing: it did not send a 100 continue so it should not be required to read a request body, regardless of transfer encoding or content-length.

Expected:

mod_proxy_http reverse proxy should not send 100-continue to a client unless the backend does.
Comment 1 Jay R. Wren 2016-11-11 22:28:03 UTC 
Created attachment 34438 [details]
in the reverse proxy case, if request had Expects: 100-continue, delay writing 100 continue response until backend has sent 100-continue response
Comment 2 Yann Ylavic 2016-11-16 00:07:40 UTC 
Created attachment 34451 [details]
Forward 100-continue (and minimize race when reusing backend connections)

I proposed this patch a while ago on the dev@ list ([1]), this is an update for latest trunk, with more (though incomplete) testing.

Could you please give it a try?

[1]. https://lists.apache.org/thread.html/4e541e032b8a77ebec8248534637b47cdcd4f38af79baa5259845db0@1430360070@%3Cdev.httpd.apache.org%3E
Comment 3 Jim Jagielski 2016-11-21 20:25:22 UTC 
*** Bug 55433 has been marked as a duplicate of this bug. ***
Comment 4 Jay R. Wren 2016-12-09 03:08:26 UTC 
Yann,

I tried that patch, but I still get 503 error when expecting a 100 Continue response.
Comment 5 Michael Osipov 2018-07-04 16:04:20 UTC 
Any chance that this will be fixed? Have the very same problem from a backend Tomcat. I guess I need to drop mod_proxy and try mod_ajp or drop Apache HTTPd altogether for this.
Comment 6 Yann Ylavic 2018-07-04 21:05:01 UTC 
*** Bug 57853 has been marked as a duplicate of this bug. ***
Comment 7 Yann Ylavic 2018-07-04 21:18:54 UTC 
Sorry it's been a long time, I think we need more informations here as to the exact issue.

What exactly isn't working with the proposed patch?
Where are 100-continue or request bodies lacking or sent inappropriately, on which side (client/backend)?
IOW, can we please have a description/scenario of what is supposed to work and how, possibly with the expected request/response on both sides?
What is the configuration being tested?
Comment 8 Michael Osipov 2018-07-06 08:05:17 UTC 
(In reply to Yann Ylavic from comment #7)
> Sorry it's been a long time, I think we need more informations here as to
> the exact issue.
> 
> What exactly isn't working with the proposed patch?
> Where are 100-continue or request bodies lacking or sent inappropriately, on
> which side (client/backend)?
> IOW, can we please have a description/scenario of what is supposed to work
> and how, possibly with the expected request/response on both sides?
> What is the configuration being tested?

Hi Yann,

I can provide a full verbose log of curl(1) for Tomcat behind HTTPd with faulty behavior and direct Tomcat communication. Moreover, I can expore the httpd.conf for that offending behavior.
Comment 9 Yann Ylavic 2018-07-06 08:28:23 UTC 
Yes please do, along with the httpd error_log with LogLevel trace7.
Thanks!
Comment 10 Michael Osipov 2018-07-06 10:57:04 UTC 
Created attachment 36015 [details]
curl(1) to Tomcat directly
Comment 11 Michael Osipov 2018-07-06 10:57:28 UTC 
Created attachment 36016 [details]
curl(1) to Tomcat via HTTPd
Comment 12 Michael Osipov 2018-07-06 11:01:12 UTC 
The error log has been sent privately due to sensitive data. Looking forward to an analysis.
Comment 13 Michael Osipov 2018-07-06 11:43:39 UTC 
FYI, I have tried mod_proxy_{http,ajp} and mod_jk.
Comment 14 Yann Ylavic 2018-07-06 11:56:55 UTC 
Thanks Michael, at first glance the error_log is with mod_proxy_ajp, while attachment 34451 [details] is about mod_proxy_http (and I'd like to keep the scope there for now).
I agree that unpatched mod_proxy_http sends "100 continue" too soon (actually independently on the client and backend side).
The patch is precisely to avoid that (hop by hop 100-continue handling), did you give it a try? If yes, could I have the error_log with mod_proxy_http?
Comment 15 Michael Osipov 2018-07-06 19:19:47 UTC 
(In reply to Yann Ylavic from comment #14)
> Thanks Michael, at first glance the error_log is with mod_proxy_ajp, while
> attachment 34451 [details] is about mod_proxy_http (and I'd like to keep the
> scope there for now).
> I agree that unpatched mod_proxy_http sends "100 continue" too soon
> (actually independently on the client and backend side).
> The patch is precisely to avoid that (hop by hop 100-continue handling), did
> you give it a try? If yes, could I have the error_log with mod_proxy_http?

I agree, I have tried all possible modules with the same negative result. I will redo for you. Moreover, I will compile from trunk along with your patch and try to reproduce. Does it still apply cleanly to trunk?

Please be patient, I won't be able to test anything before 2018-07-23.

Thank you.
Comment 16 jfclere 2018-07-23 20:28:09 UTC 
Created attachment 36046 [details]
web.xml (to put in webapps/ROOT/WEB-INF/)
Comment 17 jfclere 2018-07-23 20:29:24 UTC 
Created attachment 36047 [details]
jsp to demonstrate the issue (put it in webapps/ROOT)
Comment 18 jfclere 2018-07-23 20:39:12 UTC 
when using the web.xml and red.jsp and having
<tomcat-users xmlns="http://tomcat.apache.org/xml"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
              version="1.0">
  <role rolename="manager"/>
  <user username="tomcat" password="tomcat" roles="manager"/>
</tomcat-users>
in conf/tomcat-users.xml in tomcat you can get the problem with basic authentication:
curl --verbose --anyauth -u tomcat:tomcat http://localhost:8000/read.jsp -X POST -T file -H "Content-Type: text/plain"

file needs to be big enough ~8.5M

you will get:
[jfclere@dhcp-144-173 build]$ curl --verbose --anyauth -u tomcat:tomcat http://localhost:8000/read.jsp -X POST -T toto -H "Content-Type: text/plain"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
> POST /read.jsp HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Type: text/plain
> Content-Length: 8849904
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 504 Gateway Timeout
< Date: Mon, 23 Jul 2018 20:38:21 GMT
< Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.0h-fips
< Content-Length: 247
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>504 Gateway Timeout</title>
</head><body>
<h1>Gateway Timeout</h1>
<p>The gateway did not receive a timely response
from the upstream server or application.</p>
</body></html>
* Connection #0 to host localhost left intact
Comment 19 jfclere 2018-07-24 07:32:13 UTC 
Created attachment 36048 [details]
Patch for trunkthat works for me (based on Yann one)
Comment 20 Yann Ylavic 2018-07-24 21:58:12 UTC 
Thanks Jean-Frédéric, applied in r1836588.

This is the same patch (plus a req != NULL check) rebased on latest trunk, right?
Comment 21 Michael Osipov 2018-07-25 12:44:22 UTC 
Commit on trunk does not work for me. I will send logs privately to Yann.
Comment 22 Yann Ylavic 2018-07-25 16:41:23 UTC 
Thanks for testing Michael.

Fixed in r1836648 (hopefully).
Comment 23 jfclere 2018-08-06 12:12:15 UTC 
Fixed according to my tests.
Comment 24 Michael Osipov 2018-09-27 21:03:41 UTC 
Can we please backport this to 2.4.x branch? It is not present in 2.4.35 :-(
Comment 25 Michael Osipov 2019-01-09 13:53:56 UTC 
Folks, can we please backport to 2.4.38 finally?
Comment 26 Michael Osipov 2019-01-21 20:08:49 UTC 
This also blocks CONNECTORS-1564.
Comment 27 Yann Ylavic 2019-02-12 08:45:43 UTC 
Proposed for backport to 2.4.x (r1853409), let's see what other committers think, notably if it's too much of a change for the stable branch. If so, this will have to wait for next 2.6/3.0...
Comment 28 Yann Ylavic 2019-02-12 09:15:57 UTC 
Also, testing the tomcat CONNECTORS-1564 case with the final patch ([1]) would be very welcome ;)

[1] http://people.apache.org/~ylavic/patches/httpd-2.4.x-forward_100_continue.patch
Comment 29 Yann Ylavic 2019-02-12 09:25:53 UTC 
(In reply to Yann Ylavic from comment #28)
> tomcat CONNECTORS-1564

Looks like it's Solr actually, but all tests are welcome.
Comment 30 Michael Osipov 2019-02-13 18:25:45 UTC 
(In reply to Yann Ylavic from comment #27)
> Proposed for backport to 2.4.x (r1853409), let's see what other committers
> think, notably if it's too much of a change for the stable branch. If so,
> this will have to wait for next 2.6/3.0...

Yann, I will happily try this patch against 2.4.x next week. It would be quite disappointing if it won't make into 2.4.x because I this isn't an improvement, but a serious bug.
Comment 31 Michael Osipov 2019-02-19 15:27:37 UTC 
Dropped patch v2 to the files directory of the FreeBSD ports, patch applied cleanly, reinstalled 2.4.38 and ran my test:

> $ curl --verbose -X POST --upload $HOME/sitex-document.3305025974265279540.zip https://blnn719x.ad001.siemens.net/ld-docgen/rest/documents --negotiate -u : -H 'Content-Type: application/zip'
> * Expire in 0 ms for 6 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 1 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 2 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> * Expire in 0 ms for 1 (transfer 0x28845000)
> *   Trying 147.54.64.19...
> * TCP_NODELAY set
> * Expire in 200 ms for 4 (transfer 0x28845000)
> * Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 443 (#0)
> * ALPN, offering http/1.1
> * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /usr/local/etc/ssl/cert.pem
>   CApath: none
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
> *  start date: May 31 13:00:16 2018 GMT
> *  expire date: May 31 13:00:16 2019 GMT
> *  subjectAltName: host "blnn719x.ad001.siemens.net" matched cert's "blnn719x.ad001.siemens.net"
> *  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
> *  SSL certificate verify ok.
> > POST /ld-docgen/rest/documents HTTP/1.1
> > Host: blnn719x.ad001.siemens.net
> > User-Agent: curl/7.64.0
> > Accept: */*
> > Content-Type: application/zip
> > Content-Length: 2056
> > Expect: 100-continue
> >
> * Expire in 1000 ms for 0 (transfer 0x28845000)
> < HTTP/1.1 401
> < Date: Tue, 19 Feb 2019 15:24:34 GMT
> < Server: Apache/2.4.38 (FreeBSD) OpenSSL/1.0.2q PHP/7.2.15 SVN/1.9.9 mod_auth_gssapi/1.6.1
> < X-Frame-Options: SAMEORIGIN
> < WWW-Authenticate: Negotiate
> < Content-Type: text/html;charset=utf-8
> < Content-Language: en
> < Content-Length: 1034
> < Connection: close
> <
> * Excess found in a non pipelined read: excess = 1034 url = /ld-docgen/rest/documents (zero-length body)
> * Closing connection 0
> * Issue another request to this URL: 'https://blnn719x.ad001.siemens.net/ld-docgen/rest/documents'
> * Hostname blnn719x.ad001.siemens.net was found in DNS cache
> *   Trying 147.54.64.19...
> * TCP_NODELAY set
> * Expire in 200 ms for 4 (transfer 0x28845000)
> * Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 443 (#1)
> * ALPN, offering http/1.1
> * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /usr/local/etc/ssl/cert.pem
>   CApath: none
> * SSL re-using session ID
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
> *  start date: May 31 13:00:16 2018 GMT
> *  expire date: May 31 13:00:16 2019 GMT
> *  subjectAltName: host "blnn719x.ad001.siemens.net" matched cert's "blnn719x.ad001.siemens.net"
> *  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
> *  SSL certificate verify ok.
> * Server auth using Negotiate with user ''
> > POST /ld-docgen/rest/documents HTTP/1.1
> > Host: blnn719x.ad001.siemens.net
> > Authorization: Negotiate YIISIgYG...
> > User-Agent: curl/7.64.0
> > Accept: */*
> > Content-Type: application/zip
> > Content-Length: 2056
> > Expect: 100-continue
> >
> * Expire in 1000 ms for 0 (transfer 0x28845000)
> * Done waiting for 100-continue
> * We are completely uploaded and fine
> < HTTP/1.1 100
> < HTTP/1.1 201
> < Date: Tue, 19 Feb 2019 15:24:35 GMT
> < Server: Apache/2.4.38 (FreeBSD) OpenSSL/1.0.2q PHP/7.2.15 SVN/1.9.9 mod_auth_gssapi/1.6.1
> < X-Frame-Options: SAMEORIGIN
> < WWW-Authenticate: Negotiate oYH1MIHyoA...
> < Location: http://blnn719x.ad001.siemens.net/ld-docgen/rest/documents/2c65479c-3e1e-4019-9c6d-215283e1e53f
> < Content-Length: 0
> <
> * Closing connection 1
> 

Hooray!

Note that SPNEGO authentication is performed by the local Tomcat and not HTTPd.
Comment 32 Yann Ylavic 2019-02-19 15:59:10 UTC 
Thanks Michael for the very appropriate test: first 401 which avoids the "100 continue" danse and thus gets forwarded with "connection: close", second 201 with the full thing.

Great!
Comment 33 Michael Osipov 2019-02-19 16:53:19 UTC 
(In reply to Yann Ylavic from comment #32)
> Thanks Michael for the very appropriate test: first 401 which avoids the
> "100 continue" danse and thus gets forwarded with "connection: close",
> second 201 with the full thing.
> 
> Great!

Thank your for the extensive patch. Looking forward to see this on 2.4.x and mod_proxy_ajp to be fixed afterwards.
Comment 34 Michael Osipov 2019-02-26 19:38:37 UTC 
I have just noticed that v4 is available of the patch. I have removed v2 from ${FILESDIR} and added v4, recompiled and reinstalled apach24 port. It works flawlessly as before with v2.

What did change? Does it affect the user?
Comment 35 Stefan Eissing 2019-02-27 08:53:38 UTC 
I do not know about v3 exactly, but v4 addresses a side effect the change had for the HTTP/2 protocol implementation.
Comment 36 s.buesing 2019-03-08 13:47:12 UTC 
We also happen to stumble accross this bug. Using Apache 2.4.38 and Tomcat 7.0.90.

Applying the patch http://people.apache.org/~ylavic/patches/httpd-2.4.x-forward_100_continue.patch did not fix the problem in our case. Is this the correct patch?
Comment 37 Yann Ylavic 2019-03-08 14:04:38 UTC 
The latest patch is: http://people.apache.org/~ylavic/patches/httpd-2.4.x-forward_100_continue-v4.patch

What's the problem in your case?
Comment 38 s.buesing 2019-03-08 14:42:51 UTC 
Thanks, I tried with v4 patch as well, still recieving a 502 bad gateway after sending a file via POST.

curl -v 'http://localhost:8080/action/forms/sprint-12/contact-page-85176' -X POST -T /tmp/test.dd 
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /action/forms/sprint-12/contact-page-85176 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 104857600
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< Via: 1.1 xyz.com:8080
* We are completely uploaded and fine
< HTTP/1.1 502 Bad Gateway
< Date: Fri, 08 Mar 2019 14:34:52 GMT
< Server: Apache
< Content-Length: 293
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
</p>
<hr>
<address>Apache Server at localhost Port 8080</address>
</body></html>
* Connection #0 to host localhost left intact


Apache logs this directly after sending the request:

[Fri Mar 08 14:32:32.808488 2019] [proxy:error] [pid 15:tid 140341990573824] (104)Connection reset by peer: [client 10.244.13.181:53536] AH01084: pass request body failed to 127.0.0.1:42180 (localhost), referer: https://xyz.com/sprint-12/contact-page
[Fri Mar 08 14:32:32.808571 2019] [proxy_http:error] [pid 15:tid 140341990573824] [client 10.244.13.181:53536] AH01097: pass request body failed to 127.0.0.1:42180 (localhost) from 10.244.13.181 (), referer: https://xyz.com/sprint-12/contact-page
Comment 39 Yann Ylavic 2019-03-08 14:57:23 UTC 
Looks like a backend error, could you please provide the error log with LogLevel trace7 and mod_dumpio configured (DumpIoInput on, DumpIoOutput on)?
Comment 40 s.buesing 2019-03-08 15:28:15 UTC 
Created attachment 36478 [details]
apache-trace7-dumpio logfile

I added the log as attachment.
If I'm really mistaking something else for this bug I'm sorry.
Comment 41 Yann Ylavic 2019-03-12 23:35:14 UTC 
Everything looks fine until:
[Fri Mar 08 15:23:06.165463 2019] [proxy:error] [pid 869:tid 140678264567552] (104)Connection reset by peer: [client 127.0.0.1:53920] AH01084: pass request body failed to 127.0.0.1:42180 (localhost)
[Fri Mar 08 15:23:06.165530 2019] [proxy_http:error] [pid 869:tid 140678264567552] [client 127.0.0.1:53920] AH10154: pass request body failed to 127.0.0.1:42180 (localhost) from 127.0.0.1 () with status 502

After that httpd eats the rest of the request body (more than 100MB overall), and responds with 502. The backend really looks faulty here.
Comment 42 s.buesing 2019-03-13 12:22:02 UTC 
You're right, the backend was faulty here, we could fix the issue there and as soon as we did that it acutally worked with the unpatched version again. Sorry for the noise.
Comment 43 Michael Osipov 2019-04-02 15:22:09 UTC 
Just tried to apply v5 to 2.4.38 and 2.4.39 from the FreeBSD ports system. It is incompatible along with the other patches available there: https://github.com/freebsd/freebsd-ports/tree/master/www/apache24/files

Since 2.4.39 is already out and does not include this patch what is the status? I'd like to test out v5, but need to figure out why it is actually failing here.
Comment 44 Yann Ylavic 2019-04-02 16:03:03 UTC 
Yes, the patch did not make it to 2.4.39 which was mainly a security release, and this change is not really trivial.. Hopefully 2.4.40 is coming soon.

As for v5 ([0] below, just in case), I just tried to apply it on top of 2.4.39 (from svn) and then to apply the freebsd patches from your link, but got no conflict. What are the issues you observe exactly?

[0] http://people.apache.org/~ylavic/patches/httpd-2.4.x-forward_100_continue-v5.patch
Comment 45 Michael Osipov 2019-04-03 13:07:35 UTC 
OK, I have it now. For some reason the patch hunk against CHANGES and the docs fail to apply. I have removed those hunks and the patch applied cleanly now.

Here is the output from curl:

> + curl --verbose --http1.1 --anyauth -u : -f -s --upload-file target/lda-docgen-webapp-0.2-SNAPSHOT-backend-dev-anon.war -o manager-response.txt 'https://sitex-ldadw.ad001.siemens.net/backend-dev-anon/manager-1/text/deploy?path=/backend-dev-anon&update=false&version=009'
> * Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
> *   Trying 147.54.64.149...
> * TCP_NODELAY set
> * Connected to sitex-ldadw.ad001.siemens.net (147.54.64.149) port 443 (#0)
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: /usr/local/etc/ssl/cert.pem
>   CApath: none
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [122 bytes data]
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> { [25 bytes data]
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> { [6195 bytes data]
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> { [520 bytes data]
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> { [52 bytes data]
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> } [52 bytes data]
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=DE; O=Siemens; OU=LDA DW; CN=sitex-ldadw.ad001.siemens.net
> *  start date: Mar 19 13:10:13 2019 GMT
> *  expire date: Mar 19 13:10:13 2020 GMT
> *  subjectAltName: host "sitex-ldadw.ad001.siemens.net" matched cert's "sitex-ldadw.ad001.siemens.net"
> *  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
> *  SSL certificate verify ok.
> } [5 bytes data]
> > PUT /backend-dev-anon/manager-1/text/deploy?path=/backend-dev-anon&update=false&version=009 HTTP/1.1
> > Host: sitex-ldadw.ad001.siemens.net
> > User-Agent: curl/7.64.1
> > Accept: */*
> > Content-Length: 6501669
> > Expect: 100-continue
> >
> { [5 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [297 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [297 bytes data]
> * old SSL session ID is stale, removing
> { [5 bytes data]
> < HTTP/1.1 401
> < Date: Wed, 03 Apr 2019 13:04:06 GMT
> < Server: Apache/2.4.39 (FreeBSD) OpenSSL/1.1.1a-freebsd mod_auth_gssapi/1.6.1
> < X-Frame-Options: SAMEORIGIN
> < Cache-Control: private
> < Expires: Thu, 01 Jan 1970 00:00:00 GMT
> < WWW-Authenticate: Negotiate
> < Content-Type: text/html;charset=utf-8
> < Content-Language: en
> < Content-Length: 802
> < Connection: close
> <
> * Closing connection 0
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS alert, close notify (256):
> } [2 bytes data]
> * Issue another request to this URL: 'https://sitex-ldadw.ad001.siemens.net/backend-dev-anon/manager-1/text/deploy?path=/backend-dev-anon&update=false&version=009'
> * Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com .siemens.de'
> * Hostname sitex-ldadw.ad001.siemens.net was found in DNS cache
> *   Trying 147.54.64.149...
> * TCP_NODELAY set
> * Connected to sitex-ldadw.ad001.siemens.net (147.54.64.149) port 443 (#1)
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: /usr/local/etc/ssl/cert.pem
>   CApath: none
> * SSL re-using session ID
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [680 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [128 bytes data]
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> { [21 bytes data]
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> { [52 bytes data]
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> } [52 bytes data]
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=DE; O=Siemens; OU=LDA DW; CN=sitex-ldadw.ad001.siemens.net
> *  start date: Mar 19 13:10:13 2019 GMT
> *  expire date: Mar 19 13:10:13 2020 GMT
> *  subjectAltName: host "sitex-ldadw.ad001.siemens.net" matched cert's "sitex-ldadw.ad001.siemens.net"
> *  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
> *  SSL certificate verify ok.
> * Server auth using Negotiate with user ''
> } [5 bytes data]
> > PUT /backend-dev-anon/manager-1/text/deploy?path=/backend-dev-anon&update=false&version=009 HTTP/1.1
> > Host: sitex-ldadw.ad001.siemens.net
> > Authorization: Negotiate YIIR9QYGKwYBBQUCoIIR6TCCEe...
> > User-Agent: curl/7.64.1
> > Accept: */*
> > Content-Length: 6501669
> > Expect: 100-continue
> >
> { [5 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [297 bytes data]
> * old SSL session ID is stale, removing
> * Done waiting for 100-continue
> } [5 bytes data]
> < HTTP/1.1 100
> } [5 bytes data]
> * We are completely uploaded and fine
> { [5 bytes data]
> < HTTP/1.1 200
> < Date: Wed, 03 Apr 2019 13:04:09 GMT
> < Server: Apache/2.4.39 (FreeBSD) OpenSSL/1.1.1a-freebsd mod_auth_gssapi/1.6.1
> < X-Frame-Options: SAMEORIGIN
> < Cache-Control: private
> < Expires: Thu, 01 Jan 1970 00:00:00 GMT
> < WWW-Authenticate: Negotiate oYH1MIHyoAMKAQChCwYJKoZIhvcSA...
> < X-Content-Type-Options: nosniff
> < Content-Type: text/plain;charset=utf-8
> < Transfer-Encoding: chunked
> <
> { [78 bytes data]
> * Connection #1 to host sitex-ldadw.ad001.siemens.net left intact
> * Closing connection 1

Looking forward to 2.4.40...
Comment 46 Michael Osipov 2019-05-27 13:36:18 UTC 
I can see in Subversion that this patch has several +1 votes. Is it scheduled for 2.4.40 then?
Comment 47 Yann Ylavic 2019-05-27 20:22:46 UTC 
Yes, likely scheduled for 2.4.40, missing one vote only.
Please note that the patch has been updated to v6 to resolve some conflicts from unrelated changes already merged in 2.4.40.
Comment 48 Michael Osipov 2019-05-27 20:36:16 UTC 
(In reply to Yann Ylavic from comment #47)
> Yes, likely scheduled for 2.4.40, missing one vote only.
> Please note that the patch has been updated to v6 to resolve some conflicts
> from unrelated changes already merged in 2.4.40.

Genius, I will try the new patch tomorrow against 2.4.39.
Comment 49 Graham Leggett 2019-05-27 23:20:38 UTC 
Backported to v2.4.40.
Comment 50 Lubos Uhliarik 2020-03-03 17:34:50 UTC 
Hi all,

I'm experiencing one more issue (I'm not sure, if it is related to this bug or not). 

There is following configuration:

1) reverse proxy server with configuration:
ProxyPass /sample/ balancer://cluster/
<Proxy balancer://cluster>
  BalancerMember http://10.0.138.138:80 ping=5
  BalancerMember http://10.0.138.59:80 ping=5
</Proxy>


2) Backend server A -  10.0.138.138
- perl script - /var/www/cgi-bin/test.cgi
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_LENGTH'});

3) Backend server B -  10.0.138.59
- on this backend, we are dropping packets from reverse proxy server:
  # iptables -A INPUT -s 10.0.137.13 -j DROP
- perl script - /var/www/cgi-bin/test.cgi
#!/usr/bin/perl
print "Content-type: text/plain\r\n";
print "\r\n";
read (STDIN, $data, $ENV{'CONTENT_LENGTH'});


4) client 
# echo "abcd" > /tmp/data
# curl -X POST --data-binary '@/tmp/data' http://10.0.137.13/sample/cgi-bin/test.cgi


- when reverse proxy selects backend server B, it fails to connect to it and then fail over to backend server A. But when I collect network traffic on backend server B, invalid Content-Length is set. Network data in case of no fail-over:


"POST /cgi-bin/test.cgi HTTP/1.1
Host: 10.0.138.138
User-Agent: curl/7.65.3
Accept: */*
Content-Type: application/x-www-form-urlencoded
Expect: 100-Continue
X-Forwarded-For: 10.0.136.27
X-Forwarded-Host: 10.0.137.13
X-Forwarded-Server: fe80::f816:3eff:fe62:246
Content-Length: 5
Connection: Keep-Alive

HTTP/1.1 100 Continue

abcd
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:23:15 GMT
Server: Apache/2.4.41 (Fedora)
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8
"

vs. network data when there was a fail-over:

"POST /cgi-bin/test.cgi HTTP/1.1
Host: 10.0.138.138
User-Agent: curl/7.65.3
Accept: */*
Content-Type: application/x-www-form-urlencoded
Expect: 100-Continue
X-Forwarded-For: 10.0.136.27
X-Forwarded-Host: 10.0.137.13
X-Forwarded-Server: fe80::f816:3eff:fe62:246
Content-Length: 0
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Tue, 03 Mar 2020 12:24:18 GMT
Server: Apache/2.4.41 (Fedora)
Connection: close
Content-Type: text/plain; charset=UTF-8
"

I don't know mod_proxy very well, but I did some debugging. When reverse proxy server tries to contact backend server B, it executes ap_proxy_http_prefetch function, reads data from r->input_filters, sets temp_brigade, then stores temp_brigade into input_brigade and leaves function. Then on fail-over, it executes ap_proxy_http_prefetch function again, but no data are read and C-L is set to 0. 

1) run, without fail-over (backend server A is contacted first):
- in ap_proxy_http_prefetch function is called just once:
   - input_brigade: 
    0 | HEAP     (0x7fffc80012c8) | 5      | 0x7fffc8001188 | [abcd~] | 1
    1 | EOS      (0x7fffc8001e08) | 0      | 0x00000000 |  | n/a
   - cl_val is set to 5, old_cl_val is set to "5"

2) run, with fail-over (backend server B is contacted first, then it fails-over on backend server A)
- first call of ap_proxy_http_prefetch function is same as in case of run without fail-over
- second ap_proxy_http_prefetch call:
   - no data are read by ap_get_brigade(r->input_filters, temp_brigade,...
   - input brigade contains just EOS
   - cal_val is equal to 0, old_cl_val is set to "5", but it is afterwards overwritten:
ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_http.c:743
743             if (req->old_cl_val || req->old_te_val || bytes_read) {
(gdb) n
744                 req->old_cl_val = apr_off_t_toa(r->pool, bytes_read);

backtrace of second call:
(gdb) bt
#0  ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffc8010738, req=0x7fffc80106a8) at mod_proxy_http.c:747
#1  proxy_http_handler (r=0x7fffc8013820, worker=<optimized out>, conf=<optimized out>, url=0x7fffc8010678 "http://10.0.138.138/cgi-bin/test.cgi", proxyname=<optimized out>, 
    proxyport=<optimized out>) at mod_proxy_http.c:2063
#2  0x00007ffff74d1a2b in proxy_run_scheme_handler (r=r@entry=0x7fffc8013820, worker=0x55555568e8b8, conf=conf@entry=0x555555656560, 
    url=0x7fffc8010678 "http://10.0.138.138/cgi-bin/test.cgi", proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:3088
#3  0x00007ffff74d27ee in proxy_handler (r=0x7fffc8013820) at mod_proxy.c:1257
#4  0x000055555556db98 in ap_run_handler (r=r@entry=0x7fffc8013820) at config.c:170
#5  0x000055555556e146 in ap_invoke_handler (r=r@entry=0x7fffc8013820) at config.c:444
#6  0x00005555555a9e23 in ap_process_async_request (r=0x7fffc8013820) at http_request.c:453
#7  0x00005555555a61f3 in ap_process_http_async_connection (c=0x7fffb8001738) at http_core.c:158
#8  ap_process_http_connection (c=0x7fffb8001738) at http_core.c:252
#9  0x000055555557f048 in ap_run_process_connection (c=c@entry=0x7fffb8001738) at connection.c:42
#10 0x00007ffff74f10e9 in process_socket (thd=<optimized out>, p=<optimized out>, sock=<optimized out>, cs=0x7fffb8001690, my_child_num=<optimized out>, my_thread_num=<optimized out>)
    at event.c:1050
#11 0x00007ffff74f1a87 in worker_thread (thd=0x5555557384f8, dummy=<optimized out>) at event.c:2084
#12 0x00007ffff7d854c0 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff7cad163 in clone () from /lib64/libc.so.6
Comment 51 Yann Ylavic 2020-03-03 21:35:40 UTC 
(In reply to Lubos Uhliarik from comment #50)
This follow up issue is being handled in bug 63891, a patch for 2.4 is available there (not yet backported to 2.4.next).
Comment 52 Yann Ylavic 2023-09-14 10:06:01 UTC 
*** Bug 55433 has been marked as a duplicate of this bug. ***