Bug 5187 - RFE: try out new spamhaus PBL blocklist
Summary: RFE: try out new spamhaus PBL blocklist
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Rules (show other bugs)
Version: SVN Trunk (Latest Devel Version)
Hardware: Other other
: P5 enhancement
Target Milestone: 3.2.0
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-11-15 12:10 UTC by Justin Mason
Modified: 2007-01-17 07:06 UTC (History)
0 users



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Justin Mason 2006-11-15 12:10:31 UTC
see rulesrc/sandbox/jm/20_zen.cf in trunk.  this bug is to track
work/issues/comments...
Comment 1 Justin Mason 2006-11-27 06:52:58 UTC
Here are the current results.

0.00000  37.2752   0.0123   1.000    0.99    0.00  RCVD_IN_XBL   
0.00000  21.3403   0.0144   0.999    0.96    0.01  T_E_RCVD_IN_XBL   
0.00000  21.3403   0.0144   0.999    0.96    0.01  T_RCVD_IN_XBL   
0.00000   2.4869   0.0041   0.998    0.88    0.00  RCVD_IN_SBL   
0.00000   1.8986   0.0082   0.996    0.85    0.00  T_RCVD_IN_ZENXBL   
0.00000   1.4006   0.0041   0.997    0.85    0.00  T_RCVD_IN_ZENSBL   
0.00000  27.0548   0.2362   0.991    0.77   (n/a)  __RCVD_IN_ZEN   
0.00000  27.0416   0.2382   0.991    0.77   (n/a)  __RCVD_IN_SBL_XBL   

This is odd.  T_RCVD_IN_ZENXBL and T_RCVD_IN_ZENSBL are coming out
quite badly here.

The code looks like it should only result in *more* hits for ZENXBL:

header T_RCVD_IN_ZENXBL       eval:check_rbl('zen-notfirsthop',
'zen.spamhaus.org.', '127.0.0.[45678]')

header RCVD_IN_XBL            eval:check_rbl('sblxbl-notfirsthop',
'sbl-xbl.spamhaus.org.', '127.0.0.[456]')

I have a theory: I think the drop-off is because RCVD_IN_XBL and RCVD_IN_SBL
are using #reused results, whereas T_RCVD_IN_ZEN[SX]BL do not. If you look at
the "by contributor" data, this looks likely; compare the net-bb-* results,
which don't use #reuse, to some of the others:
http://ruleqa.spamassassin.org/20061125-r479108-n/%2FRCVD_IN_(ZEN%7CSBL%7CXBL)/detail#all

0.00000   6.5471   0.1378   0.979    0.86    0.00  RCVD_IN_SBL net-bb-fredt
0.00000   6.5338   0.1378   0.979    0.86    0.00  T_RCVD_IN_ZENSBL net-bb-fredt

0.00000   1.6660   0.0000   1.000    0.81    0.00  RCVD_IN_SBL net-theo  
0.00000   0.0005   0.0000   1.000    0.41    0.00  T_RCVD_IN_ZENSBL net-theo

in other words, I need to do a standalone mass-check with --reuse
turned *off* to get a useful comparison of this data.

also: Theo -- looks like you need to look at the resolver on that
machine ;)

finally: no sign of the PBL. odd.
Comment 2 Theo Van Dinter 2006-11-27 08:25:51 UTC
(In reply to comment #1)
> also: Theo -- looks like you need to look at the resolver on that
> machine ;)

Interesting.  DNS has been running fine on that machine, so I'm not sure why
there are basically no hits.  Postfix seems to be resolving just fine during
that time (note this is for non-spamtraps and users other than myself).
Comment 3 Daryl C. W. O'Shea 2006-11-27 08:35:06 UTC
I've seen the same sort of funkiness on well connected hosts with rbl_timeout
settings of less than 60 seconds due to the way the dynamic timeout works.
Comment 4 Justin Mason 2006-11-27 08:46:37 UTC
ah, oops.  those results are spurious.

I had checked in something last week which broke eval tests -- of course, the
DNSBL lookup stuff is implemented as a type of eval test.  that's what was going
on there; if the rule couldn't be #reused, the hitrate was tiny compared to
where it should be.  so those freqs can be entirely ignored...
Comment 5 Justin Mason 2006-12-04 05:04:54 UTC
btw, here are the results of a standalone mass-check without --reuse:

  MSECS    SPAM%     HAM%     S/O    RANK   SCORE  NAME
      0    22120    84714    0.207   0.00    0.00  (all messages)
0.00000  20.7050  79.2950    0.207   0.00    0.00  (all messages as %)
  3.912  18.8834   0.0024    1.000   0.00    3.11  RCVD_IN_XBL
  3.913  18.8879   0.0024    1.000   1.00    0.00  RCVD_IN_ZENXBL
  0.082   0.3933   0.0012    0.997   0.33    2.71  RCVD_IN_SBL
  0.082   0.3933   0.0012    0.997   0.33    0.00  RCVD_IN_ZENSBL


simple duplication of the figures there -- bar one or two mails that
probably timed out in the old sbl-xbl zone.  looks totally safe to replace,
given that.

(the ham figures are accurate, btw, not FNs.)

Still no sign of PBL, though, so I've left that in the sandbox.

svn commit -m "bug 5187: replace use of sbl-xbl.spamhaus.org with new
zen.spamhaus.org zone; leave RCVD_IN_PBL in sandbox as it's still unreleased"
rules/20_dnsbl_tests.cf rulesrc/sandbox/jm/20_zen.cf
Sending        rulesrc/sandbox/jm/20_zen.cf
Sending        rules/20_dnsbl_tests.cf
Transmitting file data ..
Committed revision 482161.


retitled bug and aimed at "Future" for when PBL appears...
Comment 6 Justin Mason 2007-01-15 10:00:46 UTC
RCVD_IN_PBL is already in the sandbox and being mass-checked, but should
be moved to rules/20_dnsbl_tests.cf before release.
Comment 7 Justin Mason 2007-01-17 07:06:23 UTC
: jm 1483...; svn commit -m "bug 5187: move RCVD_IN_PBL to main ruleset now that
it's been released; bug 5285: retire RCVD_IN_NJABL_DUL in favour of RCVD_IN_PBL"
rules/20_dnsbl_tests.cf  rulesrc/sandbox/jm/20_zen.cf
Deleting       rulesrc/sandbox/jm/20_zen.cf
Sending        rules/20_dnsbl_tests.cf
Transmitting file data .
Committed revision 497038.