Bug 2538 - address problems with Outlook forgery rules
Summary: address problems with Outlook forgery rules
Status: RESOLVED FIXED
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Rules (show other bugs)
Version: 2.60
Hardware: Other other
: P2 major
Target Milestone: 2.61
Assignee: Daniel Quinlan
URL:
Whiteboard:
Keywords:
: 1970 2096 2107 2357 2488 2527 2599 (view as bug list)
Depends on:
Blocks: 2344
  Show dependency tree
 
Reported: 2003-10-01 17:35 UTC by Daniel Quinlan
Modified: 2004-02-20 11:38 UTC (History)
8 users (show)


Attachment Type Modified Status Actions Submitter/CLA Status
proposed fix for issues 1-4 patch None Daniel Quinlan [HasCLA]
proposed fix for issues 1-4 patch None Daniel Quinlan [HasCLA]
proposed fix for issues 1-4, fixes FORGED_MUA_OUTLOOK patch None Daniel Quinlan [HasCLA]
proposed fix for issues 1-4, fixes FORGED_MUA_OUTLOOK patch None Daniel Quinlan [HasCLA]
Show Obsolete (2) Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Quinlan 2003-10-01 17:35:53 UTC 
Creating a meta-bug to include a bunch of related bugs that will be fixed
by the patch I'm working on.  This will not fix *all* of the forgery rule
problems in 2.60, but should fix most of them.  I'll only mark the ones that
are addressed by this patch as duplicates.
Comment 1 Daniel Quinlan 2003-10-01 17:45:47 UTC 
*** Bug 2107 has been marked as a duplicate of this bug. ***
Comment 2 Daniel Quinlan 2003-10-01 17:50:10 UTC 
Issue 1: Outlook IMO can also use the "dollar sign" Message-ID format (bug 2107)
Comment 3 Daniel Quinlan 2003-10-01 18:06:57 UTC 
*** Bug 2488 has been marked as a duplicate of this bug. ***
Comment 4 Daniel Quinlan 2003-10-01 18:07:34 UTC 
Issue 2: Outlook Express 4 can use 8 dollars instead of 12 for first grouping
Comment 5 Daniel Quinlan 2003-10-01 18:10:29 UTC 
*** Bug 2096 has been marked as a duplicate of this bug. ***
Comment 6 Daniel Quinlan 2003-10-01 18:11:11 UTC 
Issue 3: Outlook versions need to be specified to avoid matching on random
mail programs with "Outlook" in the same (bug 2096)
Comment 7 Daniel Quinlan 2003-10-01 18:14:13 UTC 
*** Bug 2527 has been marked as a duplicate of this bug. ***
Comment 8 Daniel Quinlan 2003-10-01 18:15:32 UTC 
*** Bug 2357 has been marked as a duplicate of this bug. ***
Comment 9 Daniel Quinlan 2003-10-01 18:16:33 UTC 
*** Bug 1970 has been marked as a duplicate of this bug. ***
Comment 10 Daniel Quinlan 2003-10-01 18:18:10 UTC 
Issue 4: MISSING_OUTLOOK_NAME triggered because __HAS_OUTLOOK_IN_MAILER does not
match "Office Outlook" (bug 2527, bug 2357, bug 1970)
Comment 11 Daniel Quinlan 2003-10-01 18:34:59 UTC 
One issue from bug 1970 will not be addressed in this bug, issue moved to
bug 2503: grupos.com.br messages may have unusable Message-ID: header.
Comment 12 Daniel Quinlan 2003-10-01 20:02:09 UTC 
Created attachment 1442 [details]
proposed fix for issues 1-4
Comment 13 Daniel Quinlan 2003-10-01 20:04:12 UTC 
Created attachment 1443 [details]
proposed fix for issues 1-4

same patch, but in unified format

I think the context one is easier to read for this patch, but since unified
is the standard here, I'll provide unified as well.
Comment 14 Daniel Quinlan 2003-10-01 21:58:40 UTC 
2.61 milestone, needs review
Comment 15 Theo Van Dinter 2003-10-01 22:30:39 UTC 
<grrr>  had the original/new backwards in diff:

<  11.684  20.8552   0.3423    0.984   0.93    0.10  MISSING_OUTLOOK_NAME
---
>  11.664  20.8552   0.2974    0.986   0.93    0.10  MISSING_OUTLOOK_NAME
4,5c4,5
<   1.922   3.4762   0.0000    1.000   0.95    2.09  FORGED_MUA_OIMO
<  14.957  27.0478   0.0047    1.000   0.98    2.95  FORGED_MUA_OUTLOOK
---
>   1.750   3.1650   0.0000    1.000   0.95    2.09  FORGED_MUA_OIMO
>  15.075  26.9600   0.3777    0.986   0.94    2.95  FORGED_MUA_OUTLOOK
Comment 16 Daniel Quinlan 2003-10-01 23:36:06 UTC 
Created attachment 1444 [details]
proposed fix for issues 1-4, fixes FORGED_MUA_OUTLOOK

This should fix the FORGED_MUA_OUTLOOK false positives.
Comment 17 Daniel Quinlan 2003-10-01 23:37:00 UTC 
Created attachment 1445 [details]
proposed fix for issues 1-4, fixes FORGED_MUA_OUTLOOK

unified version
Comment 18 Justin Mason 2003-10-02 19:06:57 UTC 
+1
Comment 19 Theo Van Dinter 2003-10-03 13:19:51 UTC 
ok, with the new patch:

<  11.684  20.8552   0.3423    0.984   0.93    0.10  MISSING_OUTLOOK_NAME
<   1.922   3.4762   0.0000    1.000   0.95    2.09  FORGED_MUA_OIMO
<  14.957  27.0478   0.0047    1.000   0.98    2.95  FORGED_MUA_OUTLOOK
---
>  11.664  20.8552   0.2974    0.986   0.95    0.10  MISSING_OUTLOOK_NAME
>   1.750   3.1650   0.0000    1.000   0.97    2.09  FORGED_MUA_OIMO
>  14.908  26.9600   0.0047    1.000   1.00    2.95  FORGED_MUA_OUTLOOK

so basically everything does worse, except MISSING_OUTLOOK_NAME, which slightly improves.
Comment 20 Daniel Quinlan 2003-10-03 13:46:54 UTC 
> so basically everything does worse, except MISSING_OUTLOOK_NAME, which
> slightly improves.

That's the idea.  The patch fixes false positive issues, so it's natural for
spam hits to drop.

The FORGED_MUA_OIMO test dropped spam hits due to bug 2107, your FPs are due
to the allowance of the "dollar sign" message ID format.  They are rare, but
do seem to happen (I have only one example out of 742 recent messages, but
I think we do need to make the change).

The FORGED_MUA_OUTLOOK test dropped spam hits due to firming up the version
numbers, etc.  It's a pretty small drop for fixing all of the false positives
that some people get.
Comment 21 Theo Van Dinter 2003-10-03 13:59:23 UTC 
Subject: Re: [SAdev]  address problems with Outlook forgery rules

On Fri, Oct 03, 2003 at 01:55:43PM -0700, bugzilla-daemon@bugzilla.spamassassin.org wrote:
> That's the idea.  The patch fixes false positive issues, so it's natural for
> spam hits to drop.

Yeah, it's just that since I had no FPs ... ;)

but I hear you.

It doesn't make anything significantly worse for me, so I'm a +1.
Comment 22 Daniel Quinlan 2003-10-03 19:45:02 UTC 
committed to branch and HEAD, closing
Comment 23 Daniel Quinlan 2003-10-15 13:06:56 UTC 
*** Bug 2599 has been marked as a duplicate of this bug. ***
Comment 24 Gary Funck 2004-02-20 20:38:37 UTC 
The Milestone above is shown as 2.61, but I'm seeing this test fail and 
misclassify ham as spam. Here's the headers in question:

X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) 
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
Resent-Message-ID: <HhqE8B.A.AzG.idzMAB@foo.org> 

It is being classified as:
 2.7 FORGED_MUA_OIMO        Forged mail pretending to be from MS Outlook IMO

The metarule in question:

# Outlook IMO (Internet Mail Only)
header __OIMO_MUA               X-Mailer =~ /Outlook IMO/
header __OIMO_MSGID             MESSAGEID =~ /^<[A-P]{26}A[AB]\.[-_\w.]
+\@\S+>$/m
meta FORGED_MUA_OIMO            (__OIMO_MUA && !__OIMO_MSGID && !
__OUTLOOK_DOLLARS_MSGID && !__UNUSABLE_MSGID)
describe FORGED_MUA_OIMO        Forged mail pretending to be from MS Outlook IMO