Bug 6220 - Evaluate Spam Eating Monkey DNSBL's
Summary: Evaluate Spam Eating Monkey DNSBL's
Status: RESOLVED WONTFIX
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Rules (show other bugs)
Version: SVN Trunk (Latest Devel Version)
Hardware: Other All
: P5 enhancement
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-10-09 10:35 UTC by Warren Togami
Modified: 2012-08-13 08:05 UTC (History)
6 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Warren Togami 2009-10-09 10:35:59 UTC
http://spameatingmonkey.com/lists.html

Put their DNSBL's and URIBL's into the sandbox with tflags nopublish so we can get statistics from weekly masscheck.
Comment 1 Mark Zealey 2011-03-23 06:18:42 UTC
This appears to be enabled now. However as we are a big ISP it is causing us issues as we issue a lot of queries. They detect this as abuse and when you abuse their system they start to return an ip of 127.0.0.255 for any query. Spamassassin then marks this as a hit and activates the score (some of which are quite high). Can you somehow change the rules to ignore any return code of 127.0.0.255?

Thanks,

Mark
Comment 2 Warren Togami 2011-03-23 06:22:56 UTC
OH NO!

This was never meant to become an active rule.  We must issue an update remove it from the active rules.
Comment 3 AXB 2011-03-23 06:25:28 UTC
(In reply to comment #1)
> This appears to be enabled now. However as we are a big ISP it is causing us
> issues as we issue a lot of queries. They detect this as abuse and when you
> abuse their system they start to return an ip of 127.0.0.255 for any query.
> Spamassassin then marks this as a hit and activates the score (some of which
> are quite high). Can you somehow change the rules to ignore any return code of
> 127.0.0.255?
> 
> Thanks,
> 
> Mark


Till an update is released, please set the rule's score to 0 (zero) to disable these queries
Comment 4 Mark Zealey 2011-03-23 06:27:01 UTC
Yes we have of course disabled it now. Even if this is not meant to be put live, could you still look at the disabling of a return result of 127.0.0.255?

Thanks,

Mark
Comment 5 Warren Togami 2011-03-23 06:30:43 UTC
OK, something is seriously wrong here.

http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/smf/30_bug_6220_sem.cf

This file begins with #testrules but all of its rules became auto-promoted into
the active ruleset.

How could this happen?
Comment 6 AXB 2011-03-23 06:32:35 UTC
(In reply to comment #4)
> Yes we have of course disabled it now. Even if this is not meant to be put
> live, could you still look at the disabling of a return result of 127.0.0.255?
> 
> Thanks,
> 
> Mark

This code is often used by BLs to warn admins that the are over query
threshold.
To disable it would break those "alerts".
Comment 7 Mark Zealey 2011-03-23 06:36:14 UTC
I agree it should keep using this, however if this code is returned spamassassin should not treat it as if the given domain is in the blacklist. All our messages suddenly started getting scores of around 6 or 7 as that's the combined total if you hit all the uribl's in there.

Mark
Comment 8 Warren Togami 2011-03-23 06:49:44 UTC
It is a VERY SERIOUS error that these network rules were auto-promoted to the sa-update channel.  We need to rectify this quickly, then figure out what went wrong for the auto-promotion backend to promote an entire file marked "#testrules".

127.0.0.255 is a separate issue entirely.  It is FULLY INTENDED for DNSBL's to cause false positive hits when you are improperly querying them.  It is the ONLY way people will pay attention and fix configuration problems.

http://www.spamtips.org/2011/01/usage-limits-of-spamassassin-network.html
Under normal circumstances the sysadmins would best know the free usage limits of various network query providers.  But in this particular instance it is our fault as a project that these rules were mistakenly enabled and for you to hit SEM's usage limits.
Comment 9 Warren Togami 2011-03-23 07:18:28 UTC
Bug #6560 is to investigate how rules in a "#testrules" file were auto-promoted into active.
Comment 10 Darxus 2011-03-23 12:24:13 UTC
Wouldn't it be better for everybody if, instead of intentionally causing false positives, the 127.0.0.255 values were detected by SA and used to disable the rule?
Comment 11 AXB 2011-03-23 12:27:34 UTC
(In reply to comment #10)
> Wouldn't it be better for everybody if, instead of intentionally causing false
> positives, the 127.0.0.255 values were detected by SA and used to disable the
> rule?

SA has no provision to do stuff like that but code contributions are welcome.
Comment 12 Michael Scheidell 2011-03-23 12:30:15 UTC
(In reply to comment #8)
>
> 
> 127.0.0.255 is a separate issue entirely.  It is FULLY INTENDED for DNSBL's to
> cause false positive hits when you are improperly querying them.  It is the
> ONLY way people will pay attention and fix configuration problems.
> 

then I would suggest that you never enable any rules by default that do this.
even spamhaus just denies your ip.  

when you start to get 10 and 20 min delays in inbound email, you will notice and then fix it.

actually setting this in a default SA install (even after your testing) could be considered a premeditated DOS attack, blocking all inbound email (if it scores > 6 or 7).

I vote you never enable any 'dual purpose' rules like this that punish someone.

on to another note:..   we found that for one or our smaller clients who had a sonicwall firewall.. those idiot things enable spamhaus rbl by default for all inbound port 25 connections, even if you don't pay sonicwall for email filtering.   they didn't do anything about it (didn't block it) but made the queries.

if these types of dual purpose rules are not disabled by default, the admin can find himself tracking down issues that are not his fault.
Comment 13 AXB 2011-03-23 12:34:07 UTC
(In reply to comment #12)
> (In reply to comment #8)
> >
> > 
> > 127.0.0.255 is a separate issue entirely.  It is FULLY INTENDED for DNSBL's to
> > cause false positive hits when you are improperly querying them.  It is the
> > ONLY way people will pay attention and fix configuration problems.
> > 
> 
> then I would suggest that you never enable any rules by default that do this.
> even spamhaus just denies your ip.  
> 
> when you start to get 10 and 20 min delays in inbound email, you will notice
> and then fix it.
> 
> actually setting this in a default SA install (even after your testing) could
> be considered a premeditated DOS attack, blocking all inbound email (if it
> scores > 6 or 7).
> 
> I vote you never enable any 'dual purpose' rules like this that punish someone.
> 
> on to another note:..   we found that for one or our smaller clients who had a
> sonicwall firewall.. those idiot things enable spamhaus rbl by default for all
> inbound port 25 connections, even if you don't pay sonicwall for email
> filtering.   they didn't do anything about it (didn't block it) but made the
> queries.

the RBL service on Sonicwalls is not enabled by default.
A human has to click a checkbox and "Apply"
(been there)
Comment 14 AXB 2011-03-23 12:41:58 UTC
Suggest we  move the discussion to dev mail list.

Discussing .255 is not relevant to the bug which is about rules being published erroneously.
Comment 15 D. Stussy 2011-06-03 21:00:49 UTC
Perhaps the 127.0.0.255 result should have its OWN rule for check_rbl_sub()?
[...And I don't mean setting a score to simply ignore it.]
Comment 16 Kevin A. McGrail 2011-06-27 22:28:40 UTC
(In reply to comment #15)
> Perhaps the 127.0.0.255 result should have its OWN rule for check_rbl_sub()?
> [...And I don't mean setting a score to simply ignore it.]

For bug tracking, the #testrules did not achieve the desired nopublish behavior and the sandbox file was first disabled by AXB and then re-enabled with nopublish flags by KAM because T_ rules for SEM went out in the rules tar for 3.3.2 (1104058) and in the update on or about 6/26 (1139459).

We need to replace the rules tar for 3.3.2 with a new one from the next update that fixes the T_ issue for SEM.
Comment 17 Darxus 2011-12-12 18:31:33 UTC
DNSWL was just disabled by default due to returning known wrong answers to in abuse situations (bug 6668).  Should this bug be closed as evaluated and deemed inappropriate for the same reason?  Should the handling of the 127.0.0.255 value be changed to not cause false positives?
Comment 18 Kevin A. McGrail 2011-12-12 18:49:16 UTC
(In reply to comment #17)
> DNSWL was just disabled by default due to returning known wrong answers to in
> abuse situations (bug 6668).  Should this bug be closed as evaluated and deemed
> inappropriate for the same reason?  Should the handling of the 127.0.0.255
> value be changed to not cause false positives?

I would agree.  

Until there is some RBL consensus on a "disabled" answer && code in SA to deal with it && it doesn't break the currently supported versions, there is no way an RBL that purposefully causes FPs due to overusage can be considered for default enabling.

Again, my understanding is that this is a good RBL that has promise and should be considered by Admins, though.

Are we on the same page?
Comment 19 Warren Togami 2011-12-12 19:09:10 UTC
(In reply to comment #18)
> (In reply to comment #17)
> > DNSWL was just disabled by default due to returning known wrong answers to in
> > abuse situations (bug 6668).  Should this bug be closed as evaluated and deemed
> > inappropriate for the same reason?  Should the handling of the 127.0.0.255
> > value be changed to not cause false positives?
> 
> I would agree.  
> 
> Until there is some RBL consensus on a "disabled" answer && code in SA to deal
> with it && it doesn't break the currently supported versions, there is no way
> an RBL that purposefully causes FPs due to overusage can be considered for
> default enabling.
> 
> Again, my understanding is that this is a good RBL that has promise and should
> be considered by Admins, though.
> 
> Are we on the same page?

Huh!?  Did something change significantly?

http://www.spamtips.org/2011/05/dnsbl-safety-report-5142011.html
http://www.spamtips.org/2011/01/dnsbl-safety-report-182011.html
http://www.spamtips.org/2011/01/dnsbl-safety-report-122011.html
http://mail-archives.apache.org/mod_mbox/spamassassin-dev/201102.mbox/%3C4D53A5F9.9010206@gmail.com%3E

SEMBLACK has a long history of inconsistent behavior.  It often has been measured to have a poor safety rating.  It usually has unacceptably high overlap.  Once I caught him outright copying other DNSBL's into his own.  More recently it behaved as a less safe subset of one of our other DNSBL's.  For these reasons I strongly advice folks to avoid using SEMBLACK.
Comment 20 Kevin A. McGrail 2011-12-12 19:14:54 UTC
(In reply to comment #19)
> (In reply to comment #18)
> > (In reply to comment #17)
> > > DNSWL was just disabled by default due to returning known wrong answers to in
> > > abuse situations (bug 6668).  Should this bug be closed as evaluated and deemed
> > > inappropriate for the same reason?  Should the handling of the 127.0.0.255
> > > value be changed to not cause false positives?
> > 
> > I would agree.  
> > 
> > Until there is some RBL consensus on a "disabled" answer && code in SA to deal
> > with it && it doesn't break the currently supported versions, there is no way
> > an RBL that purposefully causes FPs due to overusage can be considered for
> > default enabling.
> > 
> > Again, my understanding is that this is a good RBL that has promise and should
> > be considered by Admins, though.
> 
> Huh!?  Did something change significantly?
> 
> http://www.spamtips.org/2011/05/dnsbl-safety-report-5142011.html
> http://www.spamtips.org/2011/01/dnsbl-safety-report-182011.html
> http://www.spamtips.org/2011/01/dnsbl-safety-report-122011.html
> http://mail-archives.apache.org/mod_mbox/spamassassin-dev/201102.mbox/%3C4D53A5F9.9010206@gmail.com%3E
> 
> SEMBLACK has a long history of inconsistent behavior.  It often has been
> measured to have a poor safety rating.  It usually has unacceptably high
> overlap.  Once I caught him outright copying other DNSBL's into his own.  More
> recently it behaved as a less safe subset of one of our other DNSBL's.  For
> these reasons I strongly advice folks to avoid using SEMBLACK.

Nothing changed, just wasn't sure where we currently stood on SEM.  I think I mixed it up with Mailspike in my head.

OK, so this ticket can be closed as not being considered for inclusion due to anti-abuse DNS FPs coupled with poor safety and questions of admin conduct.
Comment 21 Darxus 2011-12-12 19:20:11 UTC
Should this be removed from the sandbox / masscheck?  Maybe test it out again in another year?  Or is it worth continuing to test?  
( trunk/rulesrc/sandbox/smf/30_bug_6220_sem.cf )
Comment 22 D. Stussy 2011-12-12 19:29:43 UTC
"Can be closed" doesn't necessarily mean "should be closed."

1)  The "anti-abuse DNS FP" issue should really be settled first.  This favors keeping the issue open.
a)  Does SA properly handle the "refused" return code from DNS resolvers?  Should we care that other software does not, even if SA correctly handles it?
b)  Is there a consensus that if a value were to be returned for "no answer due to abuse," it should be outside of 127.0.0.0/8?

2)  Poor response quality does favor closing the issue.
3)  Bad administration practice does favor closing the issue.

Despite #2 and #3, I really suggest that #1 above needs to be addressed before formally closing this issue.  If all concerned do in fact decide #1b should happen (even if done under a different issue # - e.g. 6668), then we have a noncompliant DNS list and we can put the matter to death permanently.
Comment 23 Warren Togami 2011-12-12 19:30:55 UTC
(In reply to comment #21)
> Should this be removed from the sandbox / masscheck?  Maybe test it out again
> in another year?  Or is it worth continuing to test?  
> ( trunk/rulesrc/sandbox/smf/30_bug_6220_sem.cf )

One benefit to keeping mediocre rules in the sandbox is to confirm that they really are still bad every few months.  Prior to my keeping track of safety ratings, admins everywhere just went by gut feelings when choosing which non-default DNSBL's to add to their MTA or Spamassassin.  They would often do so without the knowledge that the DNSBL they are adding is 30% the size of PBL, yet overlaps with PBL 90% of the time and somehow has more FP's.  Or in the case of UCEPROTECT people had NO IDEA how unsafe it was until we identified its high overlap with other quality rules.

Keeping them in weekly masscheck also gives them a chance to redeem themselves in the future.  Is it hurting us to keep it in weekly masscheck?
Comment 24 Kevin A. McGrail 2011-12-12 19:47:35 UTC
(In reply to comment #22)
> "Can be closed" doesn't necessarily mean "should be closed."
> 
> 1)  The "anti-abuse DNS FP" issue should really be settled first.  This favors
> keeping the issue open.

Sorry, I'm trying to close languishing tickets.  Feel free to reopen especially if there is any data to support opening the ticket, wonderful.  

But right now, I think there are a lot of strikes against SEM being implemented and no reason to keep this open.

Of course, I also take primordial joy in closing tickets.  Sort of the same joy I get in balancing my check book monthly. Don't take away my joy ;-)

> a)  Does SA properly handle the "refused" return code from DNS resolvers? 

No, because that refused code is non-standard. I guess if the RBLs want to help us implement the code AND we implement it AND we wait until the old versions are EOL, then we could implement RBLs.

> Should we care that other software does not, even if SA correctly handles it?

Yes, we should care because it can't be standardized easily.  It's a problem that doesn't have a good fix because of the use of DNS as a distributed network.

> b)  Is there a consensus that if a value were to be returned for "no answer due
> to abuse," it should be outside of 127.0.0.0/8?

No, because many places just deal with RBLs as yes/no and don't care what net block they belong to nor do they pay attention to bitwise answers.  

Some RBLs run combined and non-combined lists because of this.


And for the record, I agree with Warren on keeping it in masscheck for virtually identical reasons.
Comment 25 D. Stussy 2011-12-12 20:49:17 UTC
"No, because that refused code is non-standard." ???  Huh?  Excuse me:

It's defined in standards-track RFC documents (2929 section 2.3, 2136 section 2.2, 1035 section 4.1.1 [24 years old], 883 page 27 [28 years old]), and several other RFCs too.  RFC 1035 is marked as an accepted standard.

If you're saying that DNS lists aren't using the code, that's their problem, not SA's.  Does SA handle the code properly when presented or not?

RFC 5782 indicates that any DNS based list should return a value in 127.0.0.0/8 (even for IPv6 lists).  It is further noted that such lists, when their domains expire, often return values outside of this range when a registrar holds the domain open for sale.  Therefore, I don't see why you think that's a problem too....  Every valid DNS list will return values in 127/8 or none at all.
Comment 26 Kevin A. McGrail 2011-12-12 20:59:34 UTC
(In reply to comment #25)
> "No, because that refused code is non-standard." ???  Huh?  Excuse me:
> 
> It's defined in standards-track RFC documents (2929 section 2.3, 2136 section
> 2.2, 1035 section 4.1.1 [24 years old], 883 page 27 [28 years old]), and
> several other RFCs too.  RFC 1035 is marked as an accepted standard.
> 
> If you're saying that DNS lists aren't using the code, that's their problem,
> not SA's.  Does SA handle the code properly when presented or not?
> 
> RFC 5782 indicates that any DNS based list should return a value in 127.0.0.0/8
> (even for IPv6 lists).  It is further noted that such lists, when their domains
> expire, often return values outside of this range when a registrar holds the
> domain open for sale.  Therefore, I don't see why you think that's a problem
> too....  Every valid DNS list will return values in 127/8 or none at all.

I am referring to a "standard" regarding what answer to give back for a "refusing to answer due to over-quota".
Comment 27 AXB 2012-08-12 10:35:42 UTC
I'd like to propose removal of these test from sandboxes

( /trunk/rulesrc/sandbox/smf/30_bug_6220_sem.cf)
 
They're set to nopublish (since 2009) and while it's nice to test new BLs, it puts an unnecessaary load on weekly masschecks to keep them there for such a long time.

comments, votes please!
Comment 28 Warren Togami 2012-08-12 11:17:45 UTC
With masschecks flowing smoothly once again, now is especially the time to examine the performance of included vs. non-included DNSBL's.  But yes, I do agree it should be removed soon.

http://www.spamtips.org/2011/05/dnsbl-safety-report-5142011.html
I'd like to do a similar analysis to this given the current masscheck results, which would require 2-3 weeks with the newly expanded corpus.  However I am stretched too thin to work on this myself.  I am just suggesting that if this project intends to actually *do* anything useful with the DNSBL masscheck results, someone else should examine safety levels in a similar manner to these old reports.

I am also curious, was Mailspike ever included in the default SA rules, perhaps for 3.4.x?
Comment 29 AXB 2012-08-12 11:24:20 UTC
(In reply to comment #28)
> With masschecks flowing smoothly once again, now is especially the time to
> examine the performance of included vs. non-included DNSBL's.  But yes, I do
> agree it should be removed soon.
> 
> http://www.spamtips.org/2011/05/dnsbl-safety-report-5142011.html
> I'd like to do a similar analysis to this given the current masscheck
> results, which would require 2-3 weeks with the newly expanded corpus. 
> However I am stretched too thin to work on this myself.  I am just
> suggesting that if this project intends to actually *do* anything useful
> with the DNSBL masscheck results, someone else should examine safety levels
> in a similar manner to these old reports.

my last weekly masscheck took over 14 hours to run, I uploaded half a gig of logifiles, and next weeks's masscheck will be even larger so I don't feel like hammering BLs which we won't use.
They also produce  quite a bit of delay in processing, probably due to slow networks over the ponds.

> I am also curious, was Mailspike ever included in the default SA rules,
> perhaps for 3.4.x?

Yes. it is included.
Comment 30 Warren Togami 2012-08-12 11:27:39 UTC
If nobody is willing to do the type of analysis that I used to do, then yes, drop them immediately.
Comment 31 Kevin A. McGrail 2012-08-12 15:34:30 UTC
(In reply to comment #27)
> I'd like to propose removal of these test from sandboxes
> 
> ( /trunk/rulesrc/sandbox/smf/30_bug_6220_sem.cf)
>  
> They're set to nopublish (since 2009) and while it's nice to test new BLs,
> it puts an unnecessaary load on weekly masschecks to keep them there for
> such a long time.
> 
> comments, votes please!

+1 to comment them (not removal just to be clear).

However, should someone want to actively test and analyze them, I would immediately support that.  But we need mascheck to be quicker and this analysis isn't currently useful to the project.

With that said, though, unless SEM has a radical change in their project, I seem to remember that SEM was not even possibly under consideration.
Comment 32 Kevin A. McGrail 2012-08-12 15:35:20 UTC
(In reply to comment #30)
> If nobody is willing to do the type of analysis that I used to do, then yes,
> drop them immediately.

At the time, at least, I don't see it as the best use of our resources but I always loved your analysis.  Glad to see you on bugzilla!
Comment 33 John Hardin 2012-08-12 16:53:23 UTC
(In reply to comment #31)
> 
> +1 to comment them (not removal just to be clear).

Agreed. +1 to comment out.
Comment 34 AXB 2012-08-13 08:05:16 UTC
FTR: Rules commented out on Aug 12 2012