Bug 7735 - Meta rules need to handle missing/unrun dependencies
Summary: Meta rules need to handle missing/unrun dependencies
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Libraries (show other bugs)
Version: 3.4.2
Hardware: All All
: P2 blocker
Target Milestone: 4.0.0
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-10 19:06 UTC by Henrik Krohns
Modified: 2019-08-31 18:25 UTC (History)
2 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Henrik Krohns 2019-07-10 19:06:22 UTC
This will always hit

meta FOO !NOTEXISTING_RULE

Similarly this will always hit, if net tests are disabled

askdns __FOO ..sometest..
meta FOO !_FOO

Need to come up with a plan on what to do when meta rules have dependencies which don't exist or are disabled. We should not run the meta at all. Funnily I closed Bug 4549 just a few days ago, thinking it works already this way.

Metas should inherit any net flags from child, so we don't have to manually.
Comment 1 Henrik Krohns 2019-08-31 17:52:01 UTC
Committed net flags inheriting.

Sending        lib/Mail/SpamAssassin/Conf/Parser.pm
Transmitting file data .done
Committing transaction...
Committed revision 1866216.

Please try to monitor if anything funny happens in mass checks etc. :-)

This is seen from current ruleset:

meta SINGLETS_LOW_CONTRAST inherits tflag net, depends on DKIM_VALID
meta SPOOFED_FREEM_REPTO inherits tflag net, depends on SPF_PASS
meta NORDNS_LOW_CONTRAST inherits tflag net, depends on SPF_PASS
meta T_END_FUTURE_EMAILS inherits tflag net, depends on DKIM_SIGNED
meta __NOT_SPOOFED inherits tflag net, depends on SPF_PASS
meta TVD_SPACE_ENCODED inherits tflag net, depends on DKIM_VALID
meta T_SUBJ_BRKN_WORDNUMS inherits tflag net, depends on DKIM_SIGNED
meta NML_ADSP_CUSTOM_HIGH inherits tflag net, depends on __RESIGNER1
meta T_BTC_ORG inherits tflag net, depends on DKIM_SIGNED
meta URI_WP_HACKED_2 inherits tflag net, depends on SPF_PASS
meta STOCK_LOW_CONTRAST inherits tflag net, depends on __NOT_SPOOFED
meta FORM_LOW_CONTRAST inherits tflag net, depends on DKIM_VALID
meta URI_ONLY_MSGID_MALF inherits tflag net, depends on SPF_PASS
meta TEQF_USR_MSGID_MALF inherits tflag net, depends on SPF_PASS
meta FORM_FRAUD_3 inherits tflag net, depends on DKIM_VALID
meta __HTML_FONT_LOW_CONTRAST_MINFP inherits tflag net, depends on DKIM_VALID
meta TO_NO_BRKTS_MSFT inherits tflag net, depends on __NOT_SPOOFED
meta T_GB_FROMNAME_SPOOFED_EMAIL_IP inherits tflag net, depends on __NOT_SPOOFED
meta __SPOOFED_FREEM_REPTO inherits tflag net, depends on SPF_PASS
meta DKIM_INVALID inherits tflag net, depends on DKIM_SIGNED
meta URI_WP_HACKED inherits tflag net, depends on DKIM_VALID
meta SPOOFED_FREEMAIL_NO_RDNS inherits tflag net, depends on DKIM_VALID
meta __VIA_RESIGNER inherits tflag net, depends on __RESIGNER1
meta T_FROMNAME_SPOOFED_EMAIL inherits tflag net, depends on __RESIGNER1
meta TEQF_USR_MSGID_HEX inherits tflag net, depends on SPF_PASS
meta T_MANY_HDRS_LCASE inherits tflag net, depends on SPF_PASS
meta __SPOOFED_FREEMAIL inherits tflag net, depends on __NOT_SPOOFED
meta TEQF_USR_IMAGE inherits tflag net, depends on DKIM_VALID
meta BITCOIN_MALWARE inherits tflag net, depends on DKIM_VALID
meta __TO_EQ_FROM_USR_NN_MINFP inherits tflag net, depends on SPF_PASS
meta NML_ADSP_CUSTOM_MED inherits tflag net, depends on __RESIGNER2
meta HTML_SINGLET_MANY inherits tflag net, depends on SPF_PASS
meta FORM_FRAUD inherits tflag net, depends on DKIM_VALID
meta NML_ADSP_CUSTOM_LOW inherits tflag net, depends on __VIA_RESIGNER
meta __FORM_LOW_CONTRAST inherits tflag net, depends on __HTML_FONT_LOW_CONTRAST_MINFP
meta BODY_EMPTY inherits tflag net, depends on __NOT_SPOOFED
meta COMPENSATION inherits tflag net, depends on DKIM_VALID
meta T_PDS_FROM_2_EMAILS inherits tflag net, depends on __VIA_RESIGNER
meta FROM_MISSP_REPLYTO inherits tflag net, depends on SPF_PASS
meta BITCOIN_SPAM_05 inherits tflag net, depends on __SPOOFED_FREEMAIL
meta SPOOFED_FREEM_REPTO_RUS inherits tflag net, depends on __SPOOFED_FREEM_REPTO
meta BODY_URI_ONLY inherits tflag net, depends on SPF_PASS
meta TVD_SPACE_RATIO_MINFP inherits tflag net, depends on DKIM_VALID
meta HDRS_LCASE inherits tflag net, depends on SPF_PASS
meta SPOOFED_FREEM_REPTO_CHN inherits tflag net, depends on __NOT_SPOOFED
Comment 2 Henrik Krohns 2019-08-31 17:56:39 UTC
As a related commit, Reuse now forces reusing any net rules (thus the inherited ones also from above).

Sending        lib/Mail/SpamAssassin/Plugin/Reuse.pm
Transmitting file data .done
Committing transaction...
Committed revision 1866217.

reuse: forcing reuse of net rule RCVD_IN_MSPIKE_BL
reuse: forcing reuse of net rule URI_WP_HACKED_2
reuse: forcing reuse of net rule BODY_URI_ONLY
reuse: forcing reuse of net rule __RCVD_IN_MSPIKE_LOW
reuse: forcing reuse of net rule FORM_LOW_CONTRAST
reuse: forcing reuse of net rule ENV_AND_HDR_SPF_MATCH
reuse: forcing reuse of net rule URI_WP_HACKED
reuse: forcing reuse of net rule __SPF_RANDOM_SENDER
reuse: forcing reuse of net rule T_SUBJ_BRKN_WORDNUMS
reuse: forcing reuse of net rule __HTML_FONT_LOW_CONTRAST_MINFP
reuse: forcing reuse of net rule __FROM_FMBLA_NEWDOM28
reuse: forcing reuse of net rule NML_ADSP_CUSTOM_HIGH
reuse: forcing reuse of net rule FSL_BULK_SIG
reuse: forcing reuse of net rule COMPENSATION
reuse: forcing reuse of net rule T_END_FUTURE_EMAILS
reuse: forcing reuse of net rule FROM_MISSP_SPF_FAIL
reuse: forcing reuse of net rule __SPOOFED_FREEMAIL
reuse: forcing reuse of net rule NORDNS_LOW_CONTRAST
reuse: forcing reuse of net rule URI_ONLY_MSGID_MALF
reuse: forcing reuse of net rule TVD_SPACE_ENCODED
reuse: forcing reuse of net rule HDRS_LCASE
reuse: forcing reuse of net rule __PDS_SPF_ONLYALL
reuse: forcing reuse of net rule BODY_EMPTY
reuse: forcing reuse of net rule BITCOIN_MALWARE
reuse: forcing reuse of net rule SINGLETS_LOW_CONTRAST
reuse: forcing reuse of net rule T_PDS_FROM_2_EMAILS
reuse: forcing reuse of net rule TEQF_USR_MSGID_HEX
reuse: forcing reuse of net rule FORM_FRAUD
reuse: forcing reuse of net rule TO_EQ_FM_SPF_FAIL
reuse: forcing reuse of net rule __SPF_FULL_PASS
reuse: forcing reuse of net rule TEQF_USR_MSGID_MALF
reuse: forcing reuse of net rule __SPOOFED_FREEM_REPTO
reuse: forcing reuse of net rule SPOOFED_FREEM_REPTO_CHN
reuse: forcing reuse of net rule __VIA_RESIGNER
reuse: forcing reuse of net rule __FROM_FMBLA_NEWDOM
reuse: forcing reuse of net rule __FROM_FMBLA_NEWDOM14
reuse: forcing reuse of net rule DKIM_INVALID
reuse: forcing reuse of net rule HTML_SINGLET_MANY
reuse: forcing reuse of net rule __FROM_FMBLA_NDBLOCKED
reuse: forcing reuse of net rule __FORM_LOW_CONTRAST
reuse: forcing reuse of net rule STOCK_LOW_CONTRAST
reuse: forcing reuse of net rule __TO_EQ_FROM_USR_NN_MINFP
reuse: forcing reuse of net rule __TO_EQ_FM_DOM_SPF_FAIL
reuse: forcing reuse of net rule RCVD_IN_MSPIKE_ZBI
reuse: forcing reuse of net rule NML_ADSP_CUSTOM_MED
reuse: forcing reuse of net rule TO_NO_BRKTS_MSFT
reuse: forcing reuse of net rule T_MANY_HDRS_LCASE
reuse: forcing reuse of net rule FORM_FRAUD_3
reuse: forcing reuse of net rule SPOOFED_FREEM_REPTO_RUS
reuse: forcing reuse of net rule FROM_MISSP_REPLYTO
reuse: forcing reuse of net rule T_GB_FROMNAME_SPOOFED_EMAIL_IP
reuse: forcing reuse of net rule XPRIO
reuse: forcing reuse of net rule BITCOIN_SPAM_05
reuse: forcing reuse of net rule T_BTC_ORG
reuse: forcing reuse of net rule __TO_EQ_FM_SPF_FAIL
reuse: forcing reuse of net rule TO_EQ_FM_DOM_SPF_FAIL
reuse: forcing reuse of net rule NML_ADSP_CUSTOM_LOW
reuse: forcing reuse of net rule TVD_SPACE_RATIO_MINFP
reuse: forcing reuse of net rule TEQF_USR_IMAGE
reuse: forcing reuse of net rule RCVD_IN_MSPIKE_WL
reuse: forcing reuse of net rule __NOT_SPOOFED
reuse: forcing reuse of net rule SPOOFED_FREEM_REPTO
Comment 3 John Hardin 2019-08-31 18:07:55 UTC
I object to many of these rules being marked as TFLAGS NET.

Your trivial examples are indeed valid, but for more-complex rules where the NET rule is used simply as an exclusion to reduce FPs (rather than a critical component of the logical rule itself), the effect of the NET rule being zero is not important, as the non-NET masscheck scores will reflect the results of that NET exclusion rule not having an effect: the score will be lower for non-NET because the S/O will be lower. The overall rule should not be *entirely disabled* in a non-NET environment.

For this reason I also dislike the inheritance being automatic.
Comment 4 Henrik Krohns 2019-08-31 18:15:15 UTC
Yes it's a bit tricky, but honestly, who uses "non-net rulesets" these days anyway? You can't even use SpamAssassin without sa-update. And those with some strange private network usage, probably don't care that much about the scoring.

Either this or we have to come up with some other way for foolproof "reuse" masschecking. Again today I notifed I was blasting DNS queries all over, due to missing reuse flags.
Comment 5 John Hardin 2019-08-31 18:21:43 UTC
(In reply to Henrik Krohns from comment #4)
> Yes it's a bit tricky, but honestly, who uses "non-net rulesets" these days
> anyway? 

True.
Comment 6 Henrik Krohns 2019-08-31 18:25:22 UTC
But if some of these rules do indeed disappear from our daily masschecks, I might revert this change to my previous code which was simply Reuse.pm doing the same thing: forcing reuse of the net-dependent metas.