Bug 62531 - Intermittent failure to use ProxySourceAddreess on outbound proxied requests
Summary: Intermittent failure to use ProxySourceAddreess on outbound proxied requests
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy (show other bugs)
Version: 2.4.6
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-07-12 00:00 UTC by Brad
Modified: 2018-07-12 00:00 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Brad 2018-07-12 00:00:15 UTC
Overview: On a proxy server host with both internal and external network devices, I want to force outbound proxy traffic onto the external network device (via the external network gateway). I'm using ProxySourceAddress with a special marker IP address to allow iptables to mark the packets for a custom ip-route table that defaults traffic route via the external gateway.

Setup: 
  CentOS 7, httpd 2.4.6
network devices: 
 eno1 with IP 10.0.0.209/24 routing via gateway 10.0.0.1 (external)
 tun0 with IP 10.8.0.10/24 routing via gateway 10.8.0.1 (internal, 10.8.0.1 is routes to public internet)
Main routing table:
 default via 10.8.0.1
 10.0.0.0/24 via 10.0.0.1

Add path to mark outbound proxied packets and redirect them to external:
- Set up simple forward proxy listening on port 8080 (allow 10.8.0.0/24)
- Set ProxySourceAddress 10.100.0.1 in httpd proxy configuration.
- ip address ad 10.100.0.1/32 dev eno1 scope host


iptables: 
  -t mangle -A OUTPUT --src 10.100.0.1 -j MARK --set-mark 1
  -t nat -A POSTROUTING --src 10.100.0.1 -j MASQUERADE


Create a second routing table to default through the external interface.
default via 10.0.0.1 table EXTERN
10.8.0.0/24 via 10.8.0.1 table EXTERN

Add an ip rule:
  ip rule add fwmark 0x1 table EXTERN

Disable reverse path filtering (net.ipv4.conf.{default,all}.rp_filter = 0).

To reproduce:
 Set browser on any internal network host to use proxy, and load some complex web pages (e.g. login and watch some videos on youtube).
 Monitor traffic on internal gateway (10.8.0.1)
 Monitor established tcp connections on httpd 

Expected:
 There should be no traffic on the internal gateway to ports 80 & 443
 All established TCP connections should have a source of 10.100.0.1

Observed:
 There is sporadic traffic on the internal gateway to ports 80 & 443 on external hosts. Traffic instigated by the browser is being proxied out through the internal gateway to the public internet.
 lsof is showing a few established connections to the public internet with source 10.0.0.209 (instead of 10.100.0.1, which is on most of them)

Note:
 lsof shows the "good" connections as type ipv6 (!) wrapping source 10.100.0.1, and "bad" connections as type ipv4 with ip 10.0.0.209.  (I tried disabling ipv6 entirely, and the observed behavior remained unchanged).  Also, it looks like the bad connections seem to all be on daemon processes that have more than one established connection - as if the first one is good, but subsequent ones are bad).

I'm open to alternate suggestions on how to approach the high-level problem of forcing proxy outbound packets through a non-default NIC.