Bug 61501 - Reverse Proxy Not Serving Content
Summary: Reverse Proxy Not Serving Content
Status: RESOLVED DUPLICATE of bug 60706
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy_http (show other bugs)
Version: 2.4.27
Hardware: PC Linux
: P3 normal with 1 vote (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-09-07 10:38 UTC by Mike Lothian
Modified: 2017-09-11 23:47 UTC (History)
1 user (show)



Attachments
httpd.conf (3.99 KB, text/plain)
2017-09-07 10:39 UTC, Mike Lothian
Details
ssl.conf (1.09 KB, text/plain)
2017-09-07 10:39 UTC, Mike Lothian
Details
ssosite.conf (1.52 KB, text/plain)
2017-09-07 10:39 UTC, Mike Lothian
Details
proxysite.conf (1.63 KB, text/plain)
2017-09-07 10:40 UTC, Mike Lothian
Details
Log with issue (trace6) (305.10 KB, text/plain)
2017-09-07 10:41 UTC, Mike Lothian
Details
Log without issue (trace6) (259.82 KB, text/plain)
2017-09-07 10:42 UTC, Mike Lothian
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Lothian 2017-09-07 10:38:41 UTC
I've setup a reverse proxy that's protected using OpenAM and using mod_ssl, mod_proxy_html and mod_openam.so (Source code: https://stash.forgerock.org/projects/OPENAM/repos/web-agents-public/browse) 

When you go to this site it correctly redirects to the OpenAM login page and redirects back to the site upon successful logon - that page then refuses to load

It will timeout based on the ProxyPass timeout values and chrome will show a “ERR_CONTENT_LENGTH_MISMATCH” (with Chrome < 61) or a blank page on other browsers

If I restart apache (sometimes several times) it will start working and remain working until its restarted

When it's in a broken state and I do a wget on an unprotected URL it receives a 200 response but doesn't download anything:

wget https://proxysite.example.com:443/master-pages/css/buttons.css
--2017-09-07 11:36:44--  https://proxysite.example.com:443/master-pages/css/buttons.css
Resolving proxysite.example.com (proxysite.example.com)... 1.2.3.4
Connecting to proxysite.example.com (proxysite.example.com)|1.2.3.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 60150 (59K) [text/css]
Saving to: ‘buttons.css’

 0% [                                                                                                                                                                                                 ] 0           --.-K/s   in 4.8s

2017-09-07 11:36:49 (0.00 B/s) - Connection closed at byte 0. Retrying.

 

I've found someone else with the same issue: https://serverfault.com/questions/870185/apache-restart-cause-proxy-httperror-pid-120502-70008partial-results-are

I'm including my configs and the working and broken logs with trace6 enabled
Comment 1 Mike Lothian 2017-09-07 10:39:07 UTC
Created attachment 35304 [details]
httpd.conf
Comment 2 Mike Lothian 2017-09-07 10:39:26 UTC
Created attachment 35305 [details]
ssl.conf
Comment 3 Mike Lothian 2017-09-07 10:39:49 UTC
Created attachment 35306 [details]
ssosite.conf
Comment 4 Mike Lothian 2017-09-07 10:40:08 UTC
Created attachment 35307 [details]
proxysite.conf
Comment 5 Mike Lothian 2017-09-07 10:41:52 UTC
Created attachment 35308 [details]
Log with issue (trace6)
Comment 6 Mike Lothian 2017-09-07 10:42:15 UTC
Created attachment 35309 [details]
Log without issue (trace6)
Comment 7 Mike Lothian 2017-09-07 10:45:36 UTC
My first comment should read mod_http not mod_html

If there's any other info required of tracing option I can enable to help pinpoint this issue please let me know
Comment 8 Ruediger Pluem 2017-09-07 12:23:29 UTC
Your backend does not deliver the content:

[Wed Sep 06 14:08:21.317841 2017] [ssl:trace4] [pid 20072:tid 139774713394944] ssl_engine_io.c(2211): [remote 4.3.2.1:443] OpenSSL: I/O error, 5 bytes expected to read on BIO#7f1f80009740 [mem: 7f1f8000f573]

Try to do a network sniff between reverse proxy and backend to see what happens on TCP layer.
Comment 9 Mike Lothian 2017-09-07 12:43:25 UTC
That seems logical, however I'm still able to go to the site that I'm proxying directly, and only a stopping and starting the apache reverse proxy fixes the issue.  I'd expect a problem with the backend to show when communicating directly or at least be fixed by doing a refresh on the browser, not a restart of the apache reverse proxy

I'll need  to get root access of the server to try and get a tcpdump. I'll attach it when I've got that done

Is there any other debugging options I can use to get more information out?
Comment 10 Mike Lothian 2017-09-11 10:49:28 UTC
The issue was introduced in the web agent module when HEAD was added in https://github.com/apache/httpd/commit/1485d64698ef816f1586e2d772c58cb9b78c788d

The bug is listed here https://bugster.forgerock.org/jira/browse/AMAGENTS-349

Their fix hasn't been open sourced yet but should be if they release the binaries as they use the CDDL licence

I've got a fix for the agent here:

https://github.com/FireBurn/web-agents-public/commit/67b27d8a77ce28ad5bee95842136050bb52577ae

I imagine other uses of ap_method_name_of might fall foul of this
Comment 11 Mike Lothian 2017-09-11 23:47:50 UTC

*** This bug has been marked as a duplicate of bug 60706 ***