Overview: On a proxy server host with both internal and external network devices, I want to force outbound proxy traffic onto the external network device (via the external network gateway). I'm using ProxySourceAddress with a special marker IP address to allow iptables to mark the packets for a custom ip-route table that defaults traffic route via the external gateway. Setup: CentOS 7, httpd 2.4.6 network devices: eno1 with IP 10.0.0.209/24 routing via gateway 10.0.0.1 (external) tun0 with IP 10.8.0.10/24 routing via gateway 10.8.0.1 (internal, 10.8.0.1 is routes to public internet) Main routing table: default via 10.8.0.1 10.0.0.0/24 via 10.0.0.1 Add path to mark outbound proxied packets and redirect them to external: - Set up simple forward proxy listening on port 8080 (allow 10.8.0.0/24) - Set ProxySourceAddress 10.100.0.1 in httpd proxy configuration. - ip address ad 10.100.0.1/32 dev eno1 scope host iptables: -t mangle -A OUTPUT --src 10.100.0.1 -j MARK --set-mark 1 -t nat -A POSTROUTING --src 10.100.0.1 -j MASQUERADE Create a second routing table to default through the external interface. default via 10.0.0.1 table EXTERN 10.8.0.0/24 via 10.8.0.1 table EXTERN Add an ip rule: ip rule add fwmark 0x1 table EXTERN Disable reverse path filtering (net.ipv4.conf.{default,all}.rp_filter = 0). To reproduce: Set browser on any internal network host to use proxy, and load some complex web pages (e.g. login and watch some videos on youtube). Monitor traffic on internal gateway (10.8.0.1) Monitor established tcp connections on httpd Expected: There should be no traffic on the internal gateway to ports 80 & 443 All established TCP connections should have a source of 10.100.0.1 Observed: There is sporadic traffic on the internal gateway to ports 80 & 443 on external hosts. Traffic instigated by the browser is being proxied out through the internal gateway to the public internet. lsof is showing a few established connections to the public internet with source 10.0.0.209 (instead of 10.100.0.1, which is on most of them) Note: lsof shows the "good" connections as type ipv6 (!) wrapping source 10.100.0.1, and "bad" connections as type ipv4 with ip 10.0.0.209. (I tried disabling ipv6 entirely, and the observed behavior remained unchanged). Also, it looks like the bad connections seem to all be on daemon processes that have more than one established connection - as if the first one is good, but subsequent ones are bad). I'm open to alternate suggestions on how to approach the high-level problem of forcing proxy outbound packets through a non-default NIC.
We have a similar use case, and ran into a different problem. I would also be interested in a different way to reach the same goal. We approached the routing a bit differently. Instead of marking packets with iptables, we're simply adding a source routing rule for a particular source address to use the internet routing table: $ ip rule ... 32764: from a.b.c.d/31 lookup internet with a matching ProxySourceAddress in the configuration. The problem we see is if the proxy wants to connect to an IPv6 address. That cannot work with an IPv4 source address. However Apache doesn't detect this. Even though the proxy destination has both IPv6 and v4 addresses, it seems to try v6 first. This doesn't work because the routing doesn't work. There are several v6 and v4 addresses; even if it tries all of them in turn, and even if it eventually works, any client that tries to connect gives up before then.