Bug 4121 - [review] Score for user defined rules become ignored
Summary: [review] Score for user defined rules become ignored
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: spamc/spamd (show other bugs)
Version: 3.0.2
Hardware: All All
: P1 major
Target Milestone: 3.0.3
Assignee: Daryl C. W. O'Shea
URL:
Whiteboard: can be committed
Keywords:
: 4062 4160 4303 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-02-05 07:39 UTC by Peter Kruty
Modified: 2005-05-03 12:03 UTC (History)
5 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status
patch patch None Daryl C. W. O'Shea [HasCLA]
additional patch patch None Theo Van Dinter [HasCLA]

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Kruty 2005-02-05 07:39:34 UTC
With fresh installation of SA 3.0.2. When I allow user_defined rules, and as
user define any rule in user_prefs, for example this one:

    header   LUBOS     Subject =~ /kukacka/i
    score    LUBOS     4.0

and then I start spamd daemon: After first check for the rule, which correctly
triggers and add "4" to the score, every following check of the same mail,
triggers the test LUBOS, but add score "1". This has something to do with
"configuration" saving. When I start spamd with -m "N" the test correctly add
score of "4" just "N" times. When I start spamd with -m "0". All works fine.
When I set $copy_config_p to "0" in spamd line 495 - so no copies of
configuration are made - everything works fine.
Comment 1 Vicki Brown 2005-02-18 13:56:17 UTC
I've been seeing the same thing (can't prove the first time worked but certainly
now getting 1 for all user-defined scores). This was happening in 3.0.1 as well,
so I upgraded in hopes that the problem would vanish.

The following are in my .spamassassin/user_prefs
 header CF_SUB_UID               Subject =~ /vlb|Vicki/i
 score CF_SUB_UID                4.0
 describe CF_SUB_UID             Subject: contains my ID
 header CF_NOT_FOR_ME            ToCc !~ /vlb\@cfcl/
 score CF_NOT_FOR_ME             3.0
 describe CF_NOT_FOR_ME          Neither To nor Cc me

Here are the headers from a piece of spam

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on cfcl.com
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.0 required=0.5 tests=CF_NOT_FOR_ME,CF_SUB_UID,
	FORGED_RCVD_HELO autolearn=no version=3.0.2
X-Spam-Report:
	*  1.0 CF_SUB_UID Subject: contains my ID
	*  1.0 CF_NOT_FOR_ME Neither To nor Cc me
	*  3.0 FORGED_RCVD_HELO Received: contains a forged HELO

We startspamd as
    spamd -d -c --pidfile=/tmp/spamd.pid

I haven't tried the -m 0 workaround
Comment 2 Vicki Brown 2005-02-18 16:49:37 UTC
spamd -m 0 
had no affect for me
Comment 3 Michael Parker 2005-02-26 15:53:38 UTC
*** Bug 4160 has been marked as a duplicate of this bug. ***
Comment 4 Justin Mason 2005-03-06 21:28:28 UTC
we've got to fix this one -- it'll be tricky though :(  user rules are hackily
implemented, internally.  however I suspect there may be a possibility of an
easy fix; it could just be something Storable-related.
Comment 5 Daryl C. W. O'Shea 2005-03-08 21:44:58 UTC
Taking bug. I believe I've got it fixed, just have to make sure I didn't break
anything.
Comment 6 Daryl C. W. O'Shea 2005-03-08 21:46:24 UTC
reassign
Comment 7 Daryl C. W. O'Shea 2005-03-08 23:11:21 UTC
Created attachment 2688 [details]
patch

The existing 'set_default_scores' code checked the wrong copy of the scores
before setting a rule's score to the default value.

This fixes the problem and doesn't appear to break anything but this is the
first time I've looked at the scoreset code so a little review wouldn't hurt.
Comment 8 Peter Kruty 2005-03-09 01:01:57 UTC
(In reply to comment #7)
> Created an attachment (id=2688) [edit]
> patch
> 
> The existing 'set_default_scores' code checked the wrong copy of the scores
> before setting a rule's score to the default value.
> 
> This fixes the problem and doesn't appear to break anything but this is the
> first time I've looked at the scoreset code so a little review wouldn't hurt.

Great. This works for me. Thanks. I patched and watch behaviour.
Comment 9 Theo Van Dinter 2005-03-09 07:29:26 UTC
It's not initially apparent to me how the patch changes anything
($conf->{scores} should always be a reference to a $conf->{scoreset}->[#]
array), but if the issue is fixed, then +1. :)

Then again, wait a minute...  I came up with a theoretical way this could
happen, but I haven't verified anything yet:

If Storable doesn't understand references, ie: $conf->{scores} =
$conf->{scoreset}->[0], the first time the rule/score is parsed, it'll get the
right score.  Then Storable will revert {scores} and {scoreset} to different
arrays.  The second time the rule/score is parsed, the score is set in
{scoreset}, and then set_default_score goes through, doesn't see the score in
{scores} (because it's not a pointer into {scoreset} anymore), and sets
{scoreset} to the default value of 1.  Then at check() time, {scores} gets
turned back into a reference and points into {scoreset}, and you get a 1.0 for
the rule.


If this is the case (and the more I think about it, the more likely it is,) then
 we'll also want to double check any other "pointer references" we use and deal
with them appropriately in M::SA::copy_config().  Alternatively, we can kill the
horrible Storable/dclone kluge thing I wrote (it seemed like a good idea at the
time) and replace it with something better.

So yes, +1 on the patch.  BTW: I'd apply it to trunk, then set the ticket
milestone to 3.0.3 in case we actually do one before 3.1 comes out.
Comment 10 Daryl C. W. O'Shea 2005-03-09 08:21:34 UTC
I've never looked at Storable, but your theory sounds reasonable to me.

I'm still waiting on my ASF account, maybe they're waiting for my birthday
tomorrow.  I'll check it into trunk when I get access.
Comment 11 Theo Van Dinter 2005-03-09 08:25:54 UTC
Subject: Re:  [review] Score for user defined rules become ignored

On Wed, Mar 09, 2005 at 08:21:34AM -0800, bugzilla-daemon@bugzilla.spamassassin.org wrote:
> I've never looked at Storable, but your theory sounds reasonable to me.

Yeah, the more I thought about it, the more I think that's what's happening.
Basically, our code goes through and saves out all of the conf one variable at
a time.  But it doesn't do any form of "do these two variables reference the
same data element" checking.  So as far as the code is concerned, {scores} and
{scoreset}->[#] are different hashes, even though they really just point at
the same place.

I'll poke at copy_config() and at least get {scores} dealt with correctly.

> I'm still waiting on my ASF account, maybe they're waiting for my birthday
> tomorrow.  I'll check it into trunk when I get access.

;)

Comment 12 Theo Van Dinter 2005-03-09 08:37:03 UTC
fyi, this patch doesn't need a vote.  it's aimed at trunk (milestone is 3.1),
which is CTR at the moment, so you can just commit it without needing a full
vote first.

after moving the ticket to a RTC release queue (such as the 3.0.3 queue), then
there'd need to be a vote for applying it.
Comment 13 Daryl C. W. O'Shea 2005-03-09 08:39:55 UTC
Yeah, that's what I had meant the vote for.  I guess that would have been
clearer if I had already committed it to trunk.
Comment 14 Theo Van Dinter 2005-03-09 08:41:22 UTC
Created attachment 2691 [details]
additional patch

modify copy_config() to deal with scores reference specially
Comment 15 Daryl C. W. O'Shea 2005-03-10 14:19:12 UTC
Just got around to trying out Theo's patch.  Looks like the scores reference
works correctly now, so I'm +1 on it.

Moving target to 3.0.3 for review of his patch (since that's what's actually
broken).
Comment 16 Daryl C. W. O'Shea 2005-03-10 14:20:24 UTC
Comment on attachment 2688 [details]
patch

I found the problem and masked it -- not good!
Comment 17 Theo Van Dinter 2005-03-11 19:10:09 UTC
*** Bug 4062 has been marked as a duplicate of this bug. ***
Comment 18 Michael Parker 2005-04-22 13:05:10 UTC
+1
Comment 19 Justin Mason 2005-04-22 13:27:58 UTC
+1

in passing, this is why using Storable to copy configs without worrying about
refs etc. isn't really helping us anyway.  ;)
Comment 20 Theo Van Dinter 2005-04-24 19:01:41 UTC
ok, committed.  r164513
Comment 21 Daryl C. W. O'Shea 2005-05-03 20:03:02 UTC
*** Bug 4303 has been marked as a duplicate of this bug. ***