Bug 7348 - SPF_PASS not relieable
Summary: SPF_PASS not relieable
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Plugins (show other bugs)
Version: 3.4.1
Hardware: All Linux
: P2 critical
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-09-01 10:21 UTC by Reindl Harald
Modified: 2018-01-28 17:02 UTC (History)
3 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Reindl Harald 2016-09-01 10:21:07 UTC
as far as i know SA is supposed to re-use the SPF-header vom spf-policyd

well, the message below has a clear spf-pass but no SPF hits in the report header and hence "whitelist_auth" did also not trigger

even if it would not re-use the "Received-SPF" header - since it was inserted by the MTA before passing the message to the milter it is 100% sure in the local resolver cache and dns-timeouts or errors are practically not possible at that stage

feels somehow like a gambling machine and "whitelist_auth" needs to be 100% relieable (not for mailchimp like in this case but in general to distinct between forged fincancial mails and real ones)
__________________________________________

Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=198.2.182.53; helo=mail53.suw15.mcsv.net; envelope-from=bounce-mc.us13_59462513.501201-checkin=thelounge.net@mail53.suw15.mcsv.net; receiver=checkin@thelounge.net
__________________________________________

-Spam-Report: Flag: No,	* -0.2 CUST_DNSWL_8_TL_N RBL:
 dnswl-aggregate.thelounge.net (No Trust)	*      [198.2.182.53 listed in
 dnswl-aggregate.thelounge.net]	* -0.4 RCVD_IN_MSPIKE_H5 RBL: Excellent
 reputation (+5)	*      [198.2.182.53 listed in wl.mailspike.net]	*  0.3
 URIBL_GREY Contains an URL listed in the URIBL greylist	*      [URIs:
 campaign-archive2.com]	*  1.0 NIXSPAM_IXHASH DIGEST: ix.dnsbl.manitu.net	*
 -0.1 CUST_DNSWL_5_ORG_N RBL: list.dnswl.org (No Trust)	*      [198.2.182.53
 listed in list.dnswl.org]	*  0.1 HEADER_FROM_DIFFERENT_DOMAINS From and
 EnvelopeFrom 2nd level mail	*      domains are different	* -0.0
 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain	*  0.5
 CUST_BODY_BEGINS_VL BODY: Begins Very Low	*  0.0 HTML_MESSAGE BODY: HTML
 included in message	*  1.5 BAYES_50 BODY: Bayes spam probability is 40 to
 60%	*      [score: 0.5000]	*  0.0 MIME_QP_LONG_LINE RAW: Quoted-printable
 line longer than 76 chars	* -0.1 DKIM_VALID Message has at least one valid
 DKIM or DK signature	*  0.1 DKIM_SIGNED Message has a DKIM or DK signature,
 not necessarily	*      valid	* -0.1 DKIM_VALID_AU Message has a valid DKIM
 or DK signature from author's	*       domain	*  1.5 IXHASH_CHECK Message
 hits one ore more IXHASH digest-sources	* -0.0 RCVD_IN_MSPIKE_WL Mailspike
 good senders	*  0.0 T_OBFU_ATTACH_MISSP No description available.	*  0.1
 BOGOFILTER_UNSURE BOGOFILTER: message is Unsure with	*     
 bogofilter-score 0.5004
X-Virus-Scanned: Yes
Comment 1 Reindl Harald 2016-09-01 10:22:33 UTC
well, and when i take that message and feed it to spamassassin now without any config changes as expected whitelist_auth triggers:

Content analysis details:   (-100.7 points, 5.5 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
-0.2 CUST_DNSWL_8_TL_N      RBL: dnswl-aggregate.thelounge.net (No Trust)
                        [198.2.182.53 listed in dnswl-aggregate.thelounge.net]
-0.1 CUST_DNSWL_5_ORG_N     RBL: list.dnswl.org (No Trust)
                            [198.2.182.53 listed in list.dnswl.org]
-0.4 RCVD_IN_MSPIKE_H5      RBL: Excellent reputation (+5)
                            [198.2.182.53 listed in wl.mailspike.net]
-100 USER_IN_SPF_WHITELIST  From: address is in the user's SPF whitelist
 0.0 SHORTCIRCUIT           Not all rules were run, due to a shortcircuited rule
 0.0 CUST_SHORTCIRCUIT      Skip tests based on whitelists/blacklists and local
                             relays
Comment 2 AXB 2016-09-01 10:29:25 UTC
(In reply to Reindl Harald from comment #0)
> as far as i know SA is supposed to re-use the SPF-header vom spf-policyd

Where is this documented?
Comment 3 Reindl Harald 2016-09-01 10:37:37 UTC
https://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Plugin_SPF.html

ignore_received_spf_header (0|1) (default: 0)

By default, to avoid unnecessary DNS lookups, the plugin will try to use the SPF results found in any Received-SPF headers it finds in the message that could only have been added by an internal relay.

Set this option to 1 to ignore any Received-SPF headers present and to have the plugin perform the SPF check itself.

Note that unless the plugin finds an identity=helo, or some unsupported identity, it will assume that the result is a mfrom SPF check result. The only identities supported are mfrom, mailfrom and helo.
Comment 4 RW 2016-09-01 11:34:52 UTC
For this to work the Received-SPF header needs to be above a trusted Received header.
Comment 5 RW 2016-09-01 11:38:52 UTC
It shouldn't prevent SPF working though. It should fall-back to doing its own lookup.
Comment 6 Reindl Harald 2016-09-01 11:44:44 UTC
it is above a trusted-received header, in fact it's on top of the message when handover to the milter happens and if that received header would not be there and trusted all the DNSBL won't work too

"It shouldn't prevent SPF working though. It should fall-back to doing its own lookup" is what i expressed with "even if it would not re-use the "Received-SPF" header - since it was inserted by the MTA before passing the message to the milter it is 100% sure in the local resolver cache and dns-timeouts or errors are practically not possible at that stage"

so there are two anchors which should make SPF_PASS successful:

* the fact that's the response is in the DNS cache
* the fact of the poliycd-header with made 
  sure it's in the DNS cache since policyd is running 
  long before milters
_________________________________

Name:   mail-gw.thelounge.net
Address: 10.0.0.19

clear_trusted_networks
trusted_networks 10.0.0.0/24
internal_networks 10.0.0.0/24

Received-SPF: Pass (sender SPF authorized) identity=mailfrom;
 client-ip=198.2.182.53; helo=mail53.suw15.mcsv.net;
 envelope-from=bounce-mc.us13_59462513.501201-checkin=thelounge.net@mail53.s
	uw15.mcsv.net; receiver=checkin@thelounge.net
Received: from mail53.suw15.mcsv.net (mail53.suw15.mcsv.net [198.2.182.53])
	by mail-gw.thelounge.net (THELOUNGE GATEWAY) with ESMTP id 3sPYXJ39pvz9c9
	for <checkin@thelounge.net>; Wed, 31 Aug 2016 20:17:52 +0200 (CEST)
Comment 7 Reindl Harald 2016-10-01 12:14:41 UTC
is there any news?

the header is pretty clear and for sure present in the milter because the spf-policyd is running long before

Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=54.240.5.3; helo=a5-3.smtp-out.eu-west-1.amazonses.com; envelope-from=010201578022b73d-db512ddc-95e7-4730-bd72-e1fb9fb0bbdf-000000@ mailer.netflix.com

another case where SPF_PASS is missing and hence "whitelist_auth *@mailer.netflix.com" not triggered but HTTPS_HTTP_MISMATCH because the so not happening shortcircuit

in that case BAYES_00 and no problem, but god beware this happens for a often forged messagetype resulting in BAYES_99 combined with other rules which is the reason you used whitelist_auth
Comment 8 Dave Jones 2018-01-28 16:43:10 UTC
If this is still a problem, this should be fixed sooner rather than later.

I have not run across a time when SPF_PASS didn't match my own Authentication-Results header being added by OpenDMARC and python-policyd-spf.
Comment 9 Reindl Harald 2018-01-28 16:51:10 UTC
well, how should that got fixed when there was no SA release for almost two years
Comment 10 Dave Jones 2018-01-28 17:02:14 UTC
Perhaps Authentication-Results is handled differently than Received-SPF.

SA 3.4.2 is coming out soon so maybe I am trying to help get attention to this issue to get it into 3.4.2 for you.