What is your mod_remoteip configuration?

<IfModule remoteip_module>
RemoteIPHeader X-Forwarded-For
RemoteIPProxiesHeader X-Forwarded-By
RemoteIPTrustedProxyList /etc/apache2/mods-available/remoteip-trusted-proxies.lst
RemoteIPInternalProxy 127.0.0.1/8
</IfModule>

The RemoteIPTrustedProxyList contains a list of Cloudfront ips (these are the ones that work with only one ip in the XF header)
and two NGINX ips (that don't work as they return multipile ips)

RemoteIPTrustedProxyList contents (abridged)

52.124.128.0/17
54.230.0.0/16
...

Are you sure, that the requests with two items in XFF actually come from one of the addresses in the trusted list? I'm asking because the access log snippet you quoted contains 1.2.3.4 as the client IP, possibly redacted by you. So is the address at the beginning of the access log line part of the trusted proxy list (eg. one of your nginx addresses)?

You could also add remoteip:trace8 to your LogLevel and check whether there's something interesting for the faling request in the logs.

Regards,

Rainer

You are correct the 1.2.3.4 is redacted and the actual ip is in the list.

Turned on trace8 and bingo - thanks!

When a single ip 1.2.3.4 is given

[Fri Jan 25 12:41:31.453912 2019] [remoteip:trace1] [pid 9517] mod_remoteip.c(404): [client 1.2.3.47:57276] Using 1.2.3.4 as client's IP by proxies 4.5.6.7
Works

When multiple: 1.2.3.4, 10.10.10.10

[Fri Jan 25 12:41:45.544435 2019] [remoteip:debug] [pid 9523] mod_remoteip.c(341): [client 4.5.6.7:57333] AH01569: RemoteIP: Header X-Forwarded-For value of 10.10.10.10 appears to be a private IP or nonsensical.  Ignored
Fails


The x-forwarded-header contains a private internal address  and this causes it to fail.
Is this correct - as I'm not sure I can stop the NGinx proxy sending the internal address as part of the forwarded for header?

As HTTP request headers can be set by any client to arbitrary values, mod_remoteip has rules from which peers it trusts the value of the remote ip header.

There are two levels of trust here:

- trusted proxies: mod_remoteip trusts any values in the header except for private addresses

- internal proxies: like trusted proxies but also accept private addresses in the header as correct

You have declared you nginx "only" as trusted, so mod_remoteip mistrusts the private address values in the header send by the nginx. Put the proxies in front of your apache, that set private addresses into the header and which you want mod_remoteip to trust into RemoteIPInternalProxy or RemoteIPInternalProxy and try again.

I myself find the names trusted versus internal confusing but it's to late to change that. Try the "Internal" variants of the directives and see if it works.

Regards,

Rainer

Thanks for that.

I have put the Ips of the Nginx proxies into  a list of RemoteIPInternalProxyList and also tried putting the list of the 'local' ips in the list.


But unfortunately it still rejects them as:

RemoteIP: Header X-Forwarded-For value of 10.0.8.65 appears to be a private IP or nonsensical.  Ignored

The same is true for a single value in the RemoteIPInternalProxy

It seems that it just won't accept XF headers with private addresses in them.

Do you build yourself? Any chance you could rebuild with a patched (debugging) version of mod_remoteip if I provide a debug patch?

I didn't build it myself - but I can give that a go next week - thanks for your assistance with this.

Have you had any more thoughts on this - or is there anything I can do to help test?

You can't *usefully* decipher XFF private addresses across networks.
{private subnet} -> {public addressing} -> {private subnet} gives you
zero useful information about the private subnet on the other side of
the public zone.

I'm guessing you have several 10. proxies (the ngnix you mentioned)
on your side of the zone that are not deciphered?

RemoteIPInternalProxy 127.0.0.1/8

will obviously not trust any 10. addresses as part of your internal
network. You need to expand the list of RemoteIPInternalProxy links
to include all internal proxies.

I'm marking as resolved since this was a configuration question in
the first place, it belongs on users@httpd.apache.org, this issue
tracker is strictly for reporting defects in ASF source code.
See http://httpd.apache.org/lists.html#http-users for obtaining peer
help with configuration.

Thanks for that and I will post over there - but I think it is a bug as none of the suggested configurations of Trusted and Internal proxies help and always raises errors when an ip in the range 10. is present in the X-Forwared-for header.

George, again, I ask if you added all the corresponding proxies for any
10. addresses *within your intranet* are present in the RemoteIPInternalProxy
and not the RemoteIPTrustedProxy lists.

Version 2.4.25 is mentioned in this report, note 2.4.30 - 2.4.33 were missing
the correct merge of the global trust list to the virtual host trust list,
see http://svn.apache.org/viewvc?view=revision&revision=1833070

Hi William Thanks for getting back.
Yes I have listed the local 10. addresses in the RemoteIPInternalProxy list and not in the RemoteIPTrustedProxy list

Traffic from proxies (listed in RemoteIPTrustedProxy) without local 10. ips in the XF header are resolving ok it is just traffic that have 10. ips in the XF header


e.g. traffic without local ips in the XF header:
[Thu Mar 07 17:34:29.305890 2019] [remoteip:trace1] [pid 466] mod_remoteip.c(423): [client 106.120.173.132:18024] Using 106.120.173.132 as client's IP by proxies 70.132.3.133

Traffic with local 10. in XF
[Thu Mar 07 17:34:07.697226 2019] [remoteip:debug] [pid 684] mod_remoteip.c(360): [client 34.243.27.172:25236] AH01569: RemoteIP: Header X-Forwarded-For value of 10.0.7.53 appears to be a private IP or nonsensical.  Ignored, referer: {redacted}

All the 10. ips are listed in the RemoteIPInternalProxy