Bug 3329 - URIDNSBL: optimization needed
Summary: URIDNSBL: optimization needed
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Rules (show other bugs)
Version: SVN Trunk (Latest Devel Version)
Hardware: Macintosh Mac OS X
: P3 normal
Target Milestone: 3.0.0
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-04-29 00:30 UTC by jon stevens
Modified: 2004-04-29 01:25 UTC (History)
0 users


Attachment Type Modified Status Actions Submitter/CLA Status
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description jon stevens 2004-04-29 00:30:48 UTC 
here is a bit of my log...notice that there is several instances of lookups for
the same domain. wouldn't it be more efficient to just look it up once and/or
cache the information locally?

Apr 29 00:21:57 takahe spamd[1997]: debug: Found Razor2 part: part=0 engine=4
ct=0 cf=0 
Apr 29 00:21:57 takahe spamd[1997]: debug: leaving helper-app run mode 
Apr 29 00:21:57 takahe spamd[1997]: debug: Razor2 results: spam? 0  highest cf
score: 0 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: netcraft.com took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: netcraft.com took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: ximian.com took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: majen.net took 2 seconds to
look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: majen.net took 2 seconds to
look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: netcraft.com took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: osdn.com took 2 seconds to
look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: slashdot.org took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: majen.net took 2 seconds to
look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: slashdot.org took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: slashdot.org took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: ximian.com took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: osdn.com took 2 seconds to
look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: ximian.com took 2 seconds
to look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: osdn.com took 2 seconds to
look up 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: queries completed: 20
started: 14 
Apr 29 00:21:57 takahe spamd[1997]: debug: URIDNSBL: queries active:  at Thu Apr
29 00:21:57 2004
Comment 1 Theo Van Dinter 2004-04-29 00:34:24 UTC 
not every ticket is priority 1.  but this should be looked at.  moving to 3.0 queue.
Comment 2 Jeff Chan 2004-04-29 01:28:51 UTC 
On behalf of my fellow SURBL DNS administrators I thank folks for noting and
addressing this.  Any reduction in DNS queries will have a large effect in
reducing network and name server traffic.  FWIW I thought URIDNSBL had some kind
of DNS lookup caching already.  The debug output looks like redundant queries
for the same message.
Comment 3 Daniel Quinlan 2004-04-29 03:01:02 UTC 
Subject: Re:  URIDNSBL: optimization needed

The DNS code has good look-up caching already so duplicate queries
should not be going out on the wire, but it is probably faster for
SpamAssassin itself to eliminate duplicates inside of the URIDNSBL code
instead.
Comment 4 Jeff Chan 2004-04-29 03:16:08 UTC 
Yes, both sound right.  Thanks for the sanity check.  :)
Comment 5 Theo Van Dinter 2004-04-29 09:21:13 UTC 
looking into this a bit, there doesn't seem to be a problem, the debugging output is just a little 
misleading.  for example:

debug: URIDNSBL: kluge.net took 1 seconds to look up
debug: URIDNSBL: kluge.net took 1 seconds to look up
debug: URIDNSBL: kluge.net took 1 seconds to look up
debug: URIDNSBL: kluge.net took 2 seconds to look up

so it looks like it's querying the same domain 4 times.  but what it's actually doing is doing a bunch of 
queries _based on_ the domain:

>> DNSBL:ws.surbl.org.:kluge.net
debug: URIDNSBL: kluge.net took 0 seconds to look up
>> DNSBL:be.surbl.org.:kluge.net
debug: URIDNSBL: kluge.net took 0 seconds to look up
>> DNSBL:sc.surbl.org.:kluge.net
debug: URIDNSBL: kluge.net took 0 seconds to look up
>> DNSBL:sbl.spamhaus.org.:8.10.80.10
debug: URIDNSBL: kluge.net took 0 seconds to look up

So we can add some more debugging output to be clearer about what is actually going on, but there's 
no efficiency issue here.

removing the 3208 block, fyi.
Comment 6 Theo Van Dinter 2004-04-29 09:25:53 UTC 
ok, I added the info to the debug output:

debug: URIDNSBL: query for kluge.net took 1 seconds to look up (ws.surbl.org.:kluge.net)
debug: URIDNSBL: query for kluge.net took 1 seconds to look up (be.surbl.org.:kluge.net)
debug: URIDNSBL: query for kluge.net took 1 seconds to look up (sc.surbl.org.:kluge.net)
debug: URIDNSBL: query for kluge.net took 2 seconds to look up (sbl.spamhaus.org.:8.10.80.10)

r10425