Bug 7888

This is a bug tracker. You should take this question to the user mailing list.

(In reply to RW from comment #1)
> This is a bug tracker. You should take this question to the user mailing
> list.

Thanks, I will do so, I was specifically told to post my question here, but feel free to delete it.

For the records:
I told him to ask on users@ or post here because it could be a bug in our rules.

(In reply to Giovanni Bechis from comment #3)
> For the records:
> I told him to ask on users@ or post here because it could be a bug in our
> rules.

I have asked on both so far, I've got a ton of people eating me alive right now, so I really just want to know what I can do to at least mitigate the issue for the time being. Our customers' mails are backing up and I've got an angry mob I need to settle

3 points does seem a bit extreme for a tracker.

The rule is 


ALL =~ /^X-(?:[a-z]{2}){1,2}-(?:EBS|(?:Tracking|Subscriber|Delivery|Customer|Campaign)-[DSU]?id):/ism

Which matches the names  of these header:

X-Edjv-Customer-Uid, X-Edjv-Campaign-Uid, X-Edjv-EBS, X-Edjv-Tracking-Did, X-Edjv-Subscriber-Uid

(In reply to RW from comment #5)
> 3 points does seem a bit extreme for a tracker.
> 
> The rule is 
> 
> 
> ALL =~
> /^X-(?:[a-z]{2}){1,2}-(?:EBS|(?:
> Tracking|Subscriber|Delivery|Customer|Campaign)-[DSU]?id):/ism
> 
> Which matches the names  of these header:
> 
> X-Edjv-Customer-Uid, X-Edjv-Campaign-Uid, X-Edjv-EBS, X-Edjv-Tracking-Did,
> X-Edjv-Subscriber-Uid

I have temporary editing the code responsible for the tracking headers and removed then entirely, hopefully this clears up the score a little, since we're otherwise fine. Just getting hit hard by that one rule.

(In reply to Byron Kleingeld from comment #4)
> (In reply to Giovanni Bechis from comment #3)
> > For the records:
> > I told him to ask on users@ or post here because it could be a bug in our
> > rules.
> 
> I have asked on both so far

I have seen no posts regarding this on the SpamAssassin Users mailing list, so I'll respond here.

(In reply to RW from comment #5)
> 3 points does seem a bit extreme for a tracker.

That's based on such headers appearing in very little ham in our corpus. If the bulk mailer doing this was a widely-used legitimate service I'd expect to see more hammy instances of it. The scored rule does have exclusions for signs in the ham we do have, and is not hitting any ham, and hits on otherwise-very-low-scoring spam, so the rule's score is fairly high. It's not, however, a poison pill.

> the software doesn't randomize any of the marketing headers for the campaigns sent with it

Such headers were fairly prevalent in a recent forged-sender-backscatter spam campaign that impacted one of my domains. Across spams from different sources, the headers *were* (apparently) random. That is historically the kind of tactic used by spammers to avoid static pattern and checksum detection tools and to pollute spam signature databases, and does not come across as something a legitimate mass mailer would do.

X-Cjqp-Delivery-Sid: 1
X-Dsyh-Delivery-Sid: 1
X-Etxr-Delivery-Sid: 1
X-Fxyn-Delivery-Sid: 1
X-Hqve-Delivery-Sid: 85
X-Kqoy-Delivery-Sid: 1
X-Lkcl-Delivery-Sid: 1
X-Lonj-Delivery-Sid: 6
X-Mw-Delivery-Sid: 18
X-Mw-Delivery-Sid: 2
X-Mw-Delivery-Sid: 37
X-Mw-Delivery-Sid: 4
X-Mw-Delivery-Sid: 698
X-toys-en-Delivery-Sid: 41
X-Vtaf-Delivery-Sid: 2
X-Zvjg-Delivery-Sid: 23

They may instead be something like a key for the bulk service client's account. If so, they chose a poor way to do it.

So part of the answer is: Byron, you're apparently using a bulk mailer service that is being actively abused by spammers, and that's having a negative impact on the reputation of your legitimate email. Work with your provider on that part of the problem.

> Just getting hit hard by that one rule.

I have reduced the score limit to 2 points, and added some further exclusions to the scored rule (which may not help in your case, it doesn't look like you are doing any of the "hammy" things our ham samples do). These changes will probably take overnight to take effect.

(In reply to John Hardin from comment #7)
> (In reply to Byron Kleingeld from comment #4)
> > (In reply to Giovanni Bechis from comment #3)
> > > For the records:
> > > I told him to ask on users@ or post here because it could be a bug in our
> > > rules.
> > 
> > I have asked on both so far
> 
> I have seen no posts regarding this on the SpamAssassin Users mailing list,
> so I'll respond here.
> 
> (In reply to RW from comment #5)
> > 3 points does seem a bit extreme for a tracker.
> 
> That's based on such headers appearing in very little ham in our corpus. If
> the bulk mailer doing this was a widely-used legitimate service I'd expect
> to see more hammy instances of it. The scored rule does have exclusions for
> signs in the ham we do have, and is not hitting any ham, and hits on
> otherwise-very-low-scoring spam, so the rule's score is fairly high. It's
> not, however, a poison pill.
> 
> > the software doesn't randomize any of the marketing headers for the campaigns sent with it
> 
> Such headers were fairly prevalent in a recent forged-sender-backscatter
> spam campaign that impacted one of my domains. Across spams from different
> sources, the headers *were* (apparently) random. That is historically the
> kind of tactic used by spammers to avoid static pattern and checksum
> detection tools and to pollute spam signature databases, and does not come
> across as something a legitimate mass mailer would do.
> 
> X-Cjqp-Delivery-Sid: 1
> X-Dsyh-Delivery-Sid: 1
> X-Etxr-Delivery-Sid: 1
> X-Fxyn-Delivery-Sid: 1
> X-Hqve-Delivery-Sid: 85
> X-Kqoy-Delivery-Sid: 1
> X-Lkcl-Delivery-Sid: 1
> X-Lonj-Delivery-Sid: 6
> X-Mw-Delivery-Sid: 18
> X-Mw-Delivery-Sid: 2
> X-Mw-Delivery-Sid: 37
> X-Mw-Delivery-Sid: 4
> X-Mw-Delivery-Sid: 698
> X-toys-en-Delivery-Sid: 41
> X-Vtaf-Delivery-Sid: 2
> X-Zvjg-Delivery-Sid: 23
> 
> They may instead be something like a key for the bulk service client's
> account. If so, they chose a poor way to do it.
> 
> So part of the answer is: Byron, you're apparently using a bulk mailer
> service that is being actively abused by spammers, and that's having a
> negative impact on the reputation of your legitimate email. Work with your
> provider on that part of the problem.
> 
> > Just getting hit hard by that one rule.
> 
> I have reduced the score limit to 2 points, and added some further
> exclusions to the scored rule (which may not help in your case, it doesn't
> look like you are doing any of the "hammy" things our ham samples do). These
> changes will probably take overnight to take effect.

I appreciate the response, I might have fudged up the sending to the mailing list, I'm not often exposing to fancy systems such as that. In the case of the mailing provider, that would be the company I work for. I assume the bulk mailing software we're using (albeit heavily modified from the original base software) could potentially be used for illicit mailing. I had manually removed those headers from the mailer code, but I am worried if there are other parts of the software that might use those headers, like bounce handling and box monitoring, regardless, knowing how the rules work now I can, if all else fails, attempt to modify how those aspects of the software work and try to get around the rules getting our mails falsely flagged. It's shame though, this software is pretty damn good at what it does, but I can absolutely see it being abused due to it's low upfront cost.

pardom my spelling and grammer, it has been a horridly long day and my brain is fried...

(In reply to John Hardin from comment #7)

> (In reply to RW from comment #5)
> > 3 points does seem a bit extreme for a tracker.
> 
> That's based on such headers appearing in very little ham in our corpus. If
> the bulk mailer doing this was a widely-used legitimate service I'd expect
> to see more hammy instances of it. The scored rule does have exclusions for
> signs in the ham we do have, 

The only exclusion is "&& !__HAVE_BOUNCE_RELAYS".  __HAVE_BOUNCE_RELAYS is test for whether any bounce relays are configured in the VBounce plug. In most set-ups it's unconditionally false. 


>  That is historically the
> kind of tactic used by spammers to avoid static pattern and checksum
> detection tools and to pollute spam signature databases,

That's more to do with the body. I don't think there's anything of that sort that would be affected by non-standard headers.

(In reply to RW from comment #10)
> (In reply to John Hardin from comment #7)
> 
> > (In reply to RW from comment #5)
> > > 3 points does seem a bit extreme for a tracker.
> > 
> > That's based on such headers appearing in very little ham in our corpus. If
> > the bulk mailer doing this was a widely-used legitimate service I'd expect
> > to see more hammy instances of it. The scored rule does have exclusions for
> > signs in the ham we do have, 
> 
> The only exclusion is "&& !__HAVE_BOUNCE_RELAYS".  __HAVE_BOUNCE_RELAYS is
> test for whether any bounce relays are configured in the VBounce plug. In
> most set-ups it's unconditionally false. 
> 
> 
> >  That is historically the
> > kind of tactic used by spammers to avoid static pattern and checksum
> > detection tools and to pollute spam signature databases,
> 
> That's more to do with the body. I don't think there's anything of that sort
> that would be affected by non-standard headers.

I'll have to agree with this, while the software is being used maliciously, it is commercial software (https://www.mailwizz.com) and this rule would punish everyone who bought the software, legitimate or otherwise. I am forced to return those headers because the software's bounce handling and box monitoring features require them to function as it reads the values to automatically unsubscribe bouncing mails, or deadmail boxes, etc etc.

I'm at a bit of an empasse now, we've taken the time to build the reputation of our servers, set up SPF and valid DKIM records and all the DNS fluff that goes around running an email marketing platform only to get punished because it's a "spammy platform" that we've heavily modified over the course of 2 years of development.

I'm open to suggestions though, while -2 spam score isn't a death sentence for our mailers and we're working with postmasters to get our mails properly routed and whilelisted and all that jazz (Effort I seriously doubt a group of spammers would go through). It just feels a bit depressing getting punished for using off-the-shelf commercial mass mailing software as a basis.

(In reply to Byron Kleingeld from comment #11)
> (In reply to RW from comment #10)
> > (In reply to John Hardin from comment #7)
> > 
> > > (In reply to RW from comment #5)
> > > > 3 points does seem a bit extreme for a tracker.
> > > 
> > > That's based on such headers appearing in very little ham in our corpus. If
> > > the bulk mailer doing this was a widely-used legitimate service I'd expect
> > > to see more hammy instances of it. The scored rule does have exclusions for
> > > signs in the ham we do have, 
> > 
> > The only exclusion is "&& !__HAVE_BOUNCE_RELAYS".  __HAVE_BOUNCE_RELAYS is
> > test for whether any bounce relays are configured in the VBounce plug. In
> > most set-ups it's unconditionally false. 
> > 
> > 
> > >  That is historically the
> > > kind of tactic used by spammers to avoid static pattern and checksum
> > > detection tools and to pollute spam signature databases,
> > 
> > That's more to do with the body. I don't think there's anything of that sort
> > that would be affected by non-standard headers.
> 
> I'll have to agree with this, while the software is being used maliciously,
> it is commercial software (https://www.mailwizz.com) and this rule would
> punish everyone who bought the software, legitimate or otherwise. I am
> forced to return those headers because the software's bounce handling and
> box monitoring features require them to function as it reads the values to
> automatically unsubscribe bouncing mails, or deadmail boxes, etc etc.
> 
> I'm at a bit of an empasse now, we've taken the time to build the reputation
> of our servers, set up SPF and valid DKIM records and all the DNS fluff that
> goes around running an email marketing platform only to get punished because
> it's a "spammy platform" that we've heavily modified over the course of 2
> years of development.
> 
> I'm open to suggestions though, while -2 spam score isn't a death sentence
> for our mailers and we're working with postmasters to get our mails properly
> routed and whilelisted and all that jazz (Effort I seriously doubt a group
> of spammers would go through). It just feels a bit depressing getting
> punished for using off-the-shelf commercial mass mailing software as a basis.

MailWizz has finally replied to my support request with a link to the following: https://kb.mailwizz.com/articles/low-score-in-spamassassin-because-of-the-rand_mktg_header-rule

It simple suggests making the rules "non-keyed" i.e using just X- and not X-xxx- I'm curious to know if this would work as they suggest though.

its funny suggestions say user@ when debate from bugzilla goes to dev@ :=)

>The rule is 
>
>ALL =~
>/^X-(?:[a-z]{2}){1,2}-(?:EBS|(?:Tracking|Subscriber|Delivery|Customer|Campaign)->[DSU]?id):/ism
>
>Which matches the names  of these header:
>
>X-Edjv-Customer-Uid, X-Edjv-Campaign-Uid, X-Edjv-EBS, X-Edjv-Tracking-Did,
>X-Edjv-Subscriber-Uid

Note the match part /(?:[a-z]{2}){1,2}/. This will match 2 or 4 letters surrounded by dashes. If you have control over the header formatting, Change Edjv to Edgvi or Edg or Ed1v or something like that, to break the 2 or 4 character pattern. Assuming those header formats are defined in a single place in the software, they should continue to link in all the places used (unless you need to fix some scanning patterns in the software also), so should continue to perform their function, while avoiding this hit due to spammers using the software.

> while the software is being used maliciously, it is commercial software
> (https://www.mailwizz.com) and this rule would punish everyone who bought
> the software, legitimate or otherwise.

I was not aware of that, I'd assumed this was a service-based situation. Thanks for that information.

> It simple suggests making the rules "non-keyed" i.e using just X- and
> not X-xxx- I'm curious to know if this would work as they suggest though.

That seems to me the best suggestion. Why they thought using randomly-named headers was a good idea is beyond me.

Rather than taking the random bit completely out, though, I suggest changing the prefix to something usefully unique but not random, like "X-Reactivemail-" (your company name). If the software depends on those headers being unique to properly process bounces et. al., that should be sufficient.

> The only exclusion is ...

There are more now.

Closing stale and apparently resolved bug.