Bug 5373 - trusted networks isn't enough for people who use forwarding services
Summary: trusted networks isn't enough for people who use forwarding services
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Libraries (show other bugs)
Version: 3.1.8
Hardware: All All
: P5 normal
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-03-08 13:48 UTC by Rob Mueller
Modified: 2011-05-01 19:57 UTC (History)
2 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status
Add trusted_host option text/plain None Rob Mueller [NoCLA]
Code to determine if RDNS name is a dialup/dsl text/plain None Rob Mueller [NoCLA]

Note You need to log in before you can comment on or make changes to this bug.
Description Rob Mueller 2007-03-08 13:48:21 UTC
The trusted_networks allows you to define the IP addresses of networks you
trust. Consider this situation:

spam-zombie -> zoneedit-mail-forwarding -> fastmail.fm-customer

So fastmail.fm runs SpamAssassin. We set our known internal networks as
trusted_networks. This means the relay analysis algorithm will use the
zoneedit-mail-forwarding IP as the -lastexternal IP for some RBL tests such as:

header RCVD_IN_XBL eval:check_rbl('sblxbl-lastexternal', 'sbl-xbl.spamhaus.org.'

This isn't very useful, we really want to scan further back to test against the
source IP. Now the way to do this is to add all the known forwarding systems to
your trusted_networks. But to do that, you'd have to know the internal IP
arrangement of every place like gmail, yahoo, hotmail, zoneedit, pobox, etc, not
really feasible.

Instead a better idea is to add some trusted_hosts to the list. In this case, we
will "trust" a set of hosts if:
1. We think all their servers are secure and zombie free
2. We know they don't have any dialup/dsl customers with RDNS that contains
their hostname

So we have a new configuration option that looks like:

trusted_host pobox.com

If when scanning back through a Received header, we see an RDNS that ends with
pobox.com, we then do a forward lookup on the name, and if it matches the IP
address in the Received header, we assume the header isn't forged, and because
we trust pobox.com servers, we set that Received header as trusted.

This means we can add trust for whole providers with one configuration option
without having to know the internal IPs of that provider.

For the moment, I haven't created a patch for this, instead I've implemented it
as hooks that replace and override existing code (see attached).

Additional options

1. The trusted_hosts configuration option has an optional second parameter.

trusted_host pobox.com srs

When srs is added, it means we know the host supports SRS forwarding, we use
that in the Mail::SpamAssassin::Plugin::SPF to change how _get_relay works so it
doesn't scan back to the first trusted host based on the new above logic. Also
in attached code.

2. With this system, you can NOT had ISPs to your trusted_hosts set because most
ISPs have correct forward/reverse DNS for their dialup/dsl hosts, which are the
main things we're trying to check against.

A solution to this would be to try and detect dialup/dsl hosts based on their
rdns hostname. This is actually possible to do fairly accurately with a bit of
regexp hackery (see attached)

If this code were used as an exception to the is_trusted_relay test, then you
could also add ISPs to the trusted_hosts list.

3. Rather than each version/user of SA having to maintain a trusted_hosts list,
a better idea would be to have a URIWL that you could query against that would
maintain known good hostnames. Then that could be maintained separately to each
version of spamassassin. No code for that yet...
Comment 1 Rob Mueller 2007-03-08 13:54:00 UTC
Created attachment 3879 [details]
Add trusted_host option

The code we use that overwrites existing SA methods to add trusted_host

Unfortunately Mail::SpamAssassin::Message::Metadata::parse_received_headers
doesn't really provide a way to easily hook in a new trust system, so I had to
reproduce the whole function with really just a small change, these lines:

    # trusted_networks matches?
    if ($in_trusted && $did_user_specify_trust && !$self->is_trusted($relay,
$trusted))
    {
      $in_trusted = 0;		# we're in deep water now
    }

It would be nice to actually change parse_received_headers so there are more
"hook" places people could get into if they want to change the overall
algorithm in the future as well...
Comment 2 Rob Mueller 2007-03-08 13:58:43 UTC
Created attachment 3880 [details]
Code to determine if RDNS name is a dialup/dsl

We use this code on our systems to determine if a machine is a dialup/dsl host
(eg more dodgy and subject to more checks like greylisting, enumeration
detection, etc). Empirically, it seems to be pretty accurate these days
assuming that the machine has any rdns at all. If not, we assume it's on the
dodgy list.
Comment 3 Justin Mason 2007-03-13 08:40:13 UTC
I'm pretty keen on this. aiming at 3.3.0
Comment 4 Justin Mason 2007-03-20 08:14:49 UTC
btw (In reply to comment #0)
> The trusted_networks allows you to define the IP addresses of networks you
> trust. Consider this situation:
> 
> spam-zombie -> zoneedit-mail-forwarding -> fastmail.fm-customer
> 
> So fastmail.fm runs SpamAssassin. We set our known internal networks as
> trusted_networks. This means the relay analysis algorithm will use the
> zoneedit-mail-forwarding IP as the -lastexternal IP for some RBL tests such as:
> 
> header RCVD_IN_XBL eval:check_rbl('sblxbl-lastexternal', 'sbl-xbl.spamhaus.org.'
> 
> This isn't very useful, we really want to scan further back to test against the
> source IP. Now the way to do this is to add all the known forwarding systems to
> your trusted_networks.

hmm, hold on.  "trusted" != "internal" -- I think you're conflating
trusted_networks and internal_networks here.
Comment 5 Daryl C. W. O'Shea 2007-03-21 23:43:21 UTC
I haven't looked at the code, but this is something I've had in mind for a
while.  I wanted to pluginize most of Received.pm first though.  I could see
this ending up in a 3.2.x release sometime in the summer though.

The SRS support part needs to end up in the SPF plugin though, it's not
something we should screw about with while determining trusted/internal though.


(In reply to comment #4)
> hmm, hold on.  "trusted" != "internal" -- I think you're conflating
> trusted_networks and internal_networks here.

Maybe, maybe not.  In his described case they'll be the same anyway.  It really
has nothing to do with adding support for using rDNS to extend trust though.
Comment 6 Daryl C. W. O'Shea 2007-03-21 23:48:07 UTC
BTW... I meant I could see this implemented in Received.pm during 3.2,
pluginizing would be 3.3.
Comment 7 Justin Mason 2007-03-22 03:43:18 UTC
I don't think it's in time to get into 3.2.0; at this stage we should consider
that release in feature-freeze mode, IMO.  3.3.0, I'm fine with.
Comment 8 Daryl C. W. O'Shea 2007-03-22 03:48:18 UTC
Yeah, not now, but maybe a small patch to 3.2.x later...

(In reply to comment #5)
> I could see this ending up in a 3.2.x release sometime in the summer though.

Comment 9 Justin Mason 2007-03-22 04:02:55 UTC
I was about to say "no", but then I realised that it doesn't affect _existing_
behaviour -- it's a new feature driven entirely by a new config keyword.  so
maybe ;)
Comment 10 Rob Mueller 2007-03-25 17:12:28 UTC
Sorry, I'm pulling in this quote from
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5374 because I think it's
more related to this and should respond here:

> FWIW, I think extending trusted_networks towards the sender further than
> internal_networks is pointless 99.99% of the time (your local DNS cache absorbs
> any penalty of a few extra DNS lookups in systems busy enough for the penalty to
> be an issue).  I recommend that people don't do it.  The only place I see
> differing trusted/internal configs is on the MSA side where you can't determine
> auth from the headers (or POPAuth).  With msa_networks now implemented it's not
> even necessary in this case anymore.

In hindsight, I think I'm agreeing with this. I think I was confused about the
use of -lastexternal, and now I'm not convinced that my patch here actually adds
anything useful to DNSBL checks.

So I went back and checked some more to see what it was helping on, and really
it seemed to be 2 cases:
1. Forwarder hosts that for some reason were on an RBL. By extended trust into
them, their IPs aren't checked, and thus when some persons forwarding service
ends up being RBL listed for some reason, suddenly a lot of their email was
being marked as spam
2. SPF checks for forwarding services that don't use SRS because I have some
other code that overrides Mail::SpamAssassin::Plugin::SPF::_get_relay to check
each relay for ->{trust_type}.

Apart from that though, I'm not sure any of this is actually a help, and I may
have been leading things up the garden path. Still, things that reduce false
positives and overall accuracy for users is always a good thing.

Rob
Comment 11 Justin Mason 2010-01-27 02:21:21 UTC
moving most remaining 3.3.0 bugs to 3.3.1 milestone
Comment 12 Justin Mason 2010-01-27 03:17:04 UTC
reassigning, too
Comment 13 Justin Mason 2010-03-23 16:34:16 UTC
moving all open 3.3.1 bugs to 3.3.2
Comment 14 Karsten Bräckelmann 2010-03-23 17:43:15 UTC
Moving back off of Security, which got changed by accident during the mass Target Milestone move.
Comment 15 Henrik Krohns 2011-05-01 19:57:10 UTC
I'd also like to see *_networks work with rdns names. Obviously this isn't happening with 3.3.. target undefined until someone wants to tackle it.