Bug 7350 - Add Mail::SpamAssassin::Plugin::HeadersEqual plugin
Summary: Add Mail::SpamAssassin::Plugin::HeadersEqual plugin
Status: RESOLVED WONTFIX
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Plugins (show other bugs)
Version: unspecified
Hardware: All All
: P2 enhancement
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-09-22 16:20 UTC by Daniel Rudolf
Modified: 2018-10-05 08:31 UTC (History)
5 users (show)


Attachment Type Modified Status Actions Submitter/CLA Status
HeadersEqual.pm application/x-perl None Daniel Rudolf [NoCLA]
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Rudolf 2016-09-22 16:20:21 UTC 
Created attachment 5404 [details]
HeadersEqual.pm

Hi,

I would like to share my (pretty simple) SA plugin I've developed recently to do a pretty basic task: Comparing message headers against each other.

Since SA doesn't support this out-of-the-box and because I think that many users want to do something like that (just search for it on Google), I would like to submit my plugin as default plugin for the next appropriate SA release. What do you guys think about that and do I have to agree to some CLA or something?

You can use the plugin to e.g. compare the various address headers of an email against each other. If you want to compare the Return-Path: and From: headers, you could do something like the following:

   loadplugin Mail::SpamAssassin::Plugin::HeadersEqual [/path/to/HeadersEqual.pm]
   header SENDER_MISMATCH eval:headers_equal("ne", "From:addr", "Return-Path:addr")
   score  SENDER_MISMATCH 0.5

As you can see, SA will increase the score by 0.5 when the From: and Return-Path: headers don't match ("ne" for "not equal"). The plugin passes all headers to Mail::SpamAssassin::PerMsgStatus->get(). This particularly allows you to append :raw, :addr and :name to header names to adjust what is compared. :addr e.g. causes SA to remove everything except the first email address from the header field.

Please note that I'm not suggesting to add this particular rule to SA's default rules (it produces many false positives), it's just an example.

You're not limited to comparing two headers, you can use a arbitrary number of headers. Here's the full syntax:

   header HEADERS_EQUAL     eval:headers_equal(header1, header2, ...)
   header HEADERS_EQUAL     eval:headers_equal("eq", header1, header2, ...)
   header HEADERS_NOT_EQUAL eval:headers_equal("ne", header1, header2, ...)

This kind of functionality is traditionally realized using regexps, however, regexps are very inflexible and expensive for this particular use case. AFAIK there's no similar plugin yet.

The plugin's code is attached. Additionally you can find it on GitHub Gist: https://gist.github.com/PhrozenByte/4af0045b6507ceb24e4231988c7d9fcf

Feedback is highly appreciated!

Cheers,
Daniel
Comment 1 Kevin A. McGrail 2018-08-25 23:43:34 UTC 
Daniel, please submit an ICLA for consideration https://www.apache.org/licenses/icla.pdf

Are you maintaining this plugin?  Overall, it sounds like this is a candidate to be documented at https://wiki.apache.org/spamassassin/CustomPlugins

Do you have any believe this helps with stopping spam/ham in real-world streams?

The Apache SpamAssassin Dev list would be a good place to discuss this.

KAM
Comment 2 Benny Pedersen 2018-08-26 03:38:34 UTC 
sadly common fails to use loadplugin in cf files :/

move the loadplugin to one that ends in pre will solve it

# cat foo.pre
loadplugin Mail::SpamAssassin::Plugin::HeadersEqual [/path/to/HeadersEqual.pm]

# cat foo.cf
header SENDER_MISMATCH eval:headers_equal("ne", "From:addr", "Return-Path:addr")
score  SENDER_MISMATCH 0.5


and lastly, score is valid in lint, but it would be nice to support score sets

like

score SENDER_MISMATCH 0.5 0.5 0.6 0.6
Comment 3 Benny Pedersen 2018-08-26 13:29:04 UTC 
ups forgot to write that foo.cf should use ifplugin endif container

else it will fail lint when plugin is disabled

# cat foo.cf
ifplugin Mail::SpamAssassin::Plugin::HeadersEqual

... plugin configs goes here

endif # Mail::SpamAssassin::Plugin::HeadersEqual


Thanks to Daniel makeing this plugin
Comment 4 RW 2018-08-26 17:05:00 UTC 
There is already an existing rule  HEADER_FROM_DIFFERENT_DOMAINS that compares the organizational domains of the From header and the envelope sender.  This is in my experience more useful in meta rules than a direct address comparison. HEADER_FROM_DIFFERENT_DOMAINS is likely to be a better rule for scoring too, but it still only scores 0.172.
Comment 5 Daniel Rudolf 2018-08-27 12:13:22 UTC 
Hi everyone,

@Kevin: Sure, I'll sign the ICLA and send it to secretary@apache.org. Maintaining this plugin shouldn't be a big deal, so, yeah, I'm willing to do so. I'm very sure it helps with stopping spam/ham in real-world streams - I'm using it on my mail servers and it greatly improved the hit-rate. However, this naturally depends on the rules that were created using this plugin. The individual circumstances of an mail server need to be taken into consideration. That's the reason why I wouldn't recommend adding particular rules, but just the plugin, allowing server admins to create their own rules.

@Benny: To be honest, I'm not sure what I'm supposed to do now. Are we talking about the plugin's docs header?

@RW: Please refer to the original post:

> Please note that I'm not suggesting to add this particular rule to SA's
> default rules (it produces many false positives), it's just an example.

Cheers,
Daniel
Comment 6 Henrik Krohns 2018-10-05 08:31:33 UTC 
Consider having evals

headers_equal
headers_not_equal

Much more logical to use, instead of remembering the strangely optional "eq"/"ne".