Bug 7269 - TXREP "txrep_whitelist_out" is not working with "auto_whitelist_distinguish_signed"
Summary: TXREP "txrep_whitelist_out" is not working with "auto_whitelist_distinguish_s...
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Plugins (show other bugs)
Version: 3.4.1
Hardware: PC Linux
: P2 normal
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-12-02 01:50 UTC by Andrew
Modified: 2015-12-15 22:16 UTC (History)
2 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew 2015-12-02 01:50:27 UTC
The ticket was opened in GitHub (https://github.com/truxoft/TxRep/issues/10) but seems to be dead there, so copying it here:

I'm trying to configure TxRep for whitelisting outgoing mail, and seems that whitelisting of outbound messages by rcpt address works ONLY if auto_whitelist_distinguish_signed = 0.

If trying to use auto_whitelist_distinguish_signed = 1 the outgoing whitelist record has stil signedby empty field too. and then this record CAN NOT be found in the database when the whitelisted address sends a message with its DKIM SIGNED.

however if I set auto_whitelist_distinguish_signed to zero I see the total SA score changes around -1.5 points after sending outbound mesage to that mail. with this option not to zero - I got only -0.1 points total score by TXREP and increasing the value of txrep_whitelist_out does not change it
Comment 1 Kevin A. McGrail 2015-12-15 19:01:35 UTC
With Ivo unavailable, I don't know that we have a huge amount of input available.  However, as a volunteer project, we do welcome patches if you have thoughts on how it should work.
Comment 2 Andrew 2015-12-15 19:42:06 UTC
Well, regarding of how it should work, here is my vision of the algoryhtm that solving this issue: 

- For the first of all the records addded by outgoing whitelist feature should be stored separately in the db and not mixed to other records.

- For example field "ip" in the case of outgoing white listing should have some personal value like "WHITEOUT" that identifies this records as outgoing whitelist record type.

- The "signedby" field fot such records should be always blank and not depend on auto_whitelist_distinguish_signed value.

- When analyzing new incoming messages and finding DB for the records we should check for the whiteout records first: try to find the records that hit the local user name (or overrided user name if set) and email fields (correspondent to currently analyzing message) AND field "ip" = WHITEOUT. such records should hit the saerch result not depending on "signedby" field and nevermind of dkim signature of the email being analyzed. The checking of the WHITEOUT records should be done separately from checking all other records (and of course when analyzing other records that are done now - such WHITEOUT records should be ignored to prevent double hits).

- WHITEOUT records scores values can be changed when analyzing new incoming and ourgoing mail as done now with generic records.

what do you think about it ?
Comment 3 Kevin A. McGrail 2015-12-15 19:49:26 UTC
According to the docs, auto_whitelist_distinguish_signed was added solely for 3.3.0 database compatibility:

 auto_whitelist_distinguish_signed
         (default: 1 - enabled)

        Used by the SQLBasedAddrList storage implementation.

        If this option is set the SQLBasedAddrList module will keep separate
        database entries for DKIM-validated e-mail addresses and for
        non-validated ones. A pre-requisite when setting this option is that
        a field awl.signedby exists in a SQL table, otherwise SQL operations
        will fail (which is why we need this option at all - for
        compatibility with pre-3.3.0 database schema). A plugin DKIM should
        also be enabled, as otherwise there is no benefit from turning on
        this option.


So if you are using 3.4.X, I'm expecting this option to be off and perhaps the database key on email is incorrect which explains the duplicate SQL errors as well.

So yes, I think the whitelisting of outbound only with this feature disabled makes sense.  It was a transitional option from what I gather.
Comment 4 Andrew 2015-12-15 20:20:37 UTC
(In reply to Kevin A. McGrail from comment #3)
> According to the docs, auto_whitelist_distinguish_signed was added solely
> for 3.3.0 database compatibility:
...
>         If this option is set the SQLBasedAddrList module will keep separate
>         database entries for DKIM-validated e-mail addresses and for
>         non-validated ones. A pre-requisite when setting this option is that
>         a field awl.signedby exists in a SQL table, otherwise SQL operations
>         will fail (which is why we need this option at all - for
>         compatibility with pre-3.3.0 database schema). A plugin DKIM should
Sorry, but I think that it means that pre-3.3.0 db doesn't have the field "signed by" and this option should be set to zero when using old scheme (turned off). So I think the documentation means "why we need this options to be TURNED OFF". By default this option is turned on (for the latest DB that has signedby field).
the DKIM separate records is a feature that helps to prevent of trashing the score for good senders when spammers use someone's "from" field but can not provide the users correct DKIM signature. when this option is not used and spammer user the email of some good user we have bad reputation for this good user just because spamnmer used his email and this is not correct, we will score the good emails with this incorrect score. For example an intruder can generate much spam and use email of a victim, the system will learn that this email is spam by txrep. then our victim writes us a good email but it is banned by txrep.
so I think this is a feature to prevent such security holes in spam protection.. and it can be turned off for old db when there were no support of this feature in txrep/awl.
Comment 5 Kevin A. McGrail 2015-12-15 20:24:56 UTC
(In reply to Andrew from comment #4)
> (In reply to Kevin A. McGrail from comment #3)
> > According to the docs, auto_whitelist_distinguish_signed was added solely
> > for 3.3.0 database compatibility:
> ...
> >         If this option is set the SQLBasedAddrList module will keep separate
> >         database entries for DKIM-validated e-mail addresses and for
> >         non-validated ones. A pre-requisite when setting this option is that
> >         a field awl.signedby exists in a SQL table, otherwise SQL operations
> >         will fail (which is why we need this option at all - for
> >         compatibility with pre-3.3.0 database schema). A plugin DKIM should
> Sorry, but I think that it means that pre-3.3.0 db doesn't have the field
> "signed by" and this option should be set to zero when using old scheme
> (turned off). So I think the documentation means "why we need this options
> to be TURNED OFF". By default this option is turned on (for the latest DB
> that has signedby field).
> the DKIM separate records is a feature that helps to prevent of trashing the
> score for good senders when spammers use someone's "from" field but can not
> provide the users correct DKIM signature. when this option is not used and
> spammer user the email of some good user we have bad reputation for this
> good user just because spamnmer used his email and this is not correct, we
> will score the good emails with this incorrect score. For example an
> intruder can generate much spam and use email of a victim, the system will
> learn that this email is spam by txrep. then our victim writes us a good
> email but it is banned by txrep.
> so I think this is a feature to prevent such security holes in spam
> protection.. and it can be turned off for old db when there were no support
> of this feature in txrep/awl.


Then have you set option to 0 when using a DB design from 3.4.1?
Comment 6 Andrew 2015-12-15 21:19:52 UTC
Yes, sure, it is working when I turn off this feature. In that case The field signedby is empty in all of the records (compatibility mode with old db). But it is better to use Txrep with this feature because of the holes that I described in previous post
Comment 7 Kevin A. McGrail 2015-12-15 21:47:44 UTC
(In reply to Andrew from comment #6)
> Yes, sure, it is working when I turn off this feature. In that case The
> field signedby is empty in all of the records (compatibility mode with old
> db). But it is better to use Txrep with this feature because of the holes
> that I described in previous post

Did you install DKIM?  I believe that's needed for the pre-db design to populate singnedby.
Comment 8 Andrew 2015-12-15 22:16:50 UTC
(In reply to Kevin A. McGrail from comment #7)
> (In reply to Andrew from comment #6)
> Did you install DKIM?  I believe that's needed for the pre-db design to
> populate singnedby.
yes, sure. and DKIM is looks like working in TxRep. The incoming mail is getting signedby fields whet I turn on this feature. BUT the outgoing white list
 scores are not recognised because outgoin mail can not have the DKIM signature of the remote address (the address that is added to the whitelist. it should be signed by the SENDER's server and its key.). the outgoing mail can only be signed by LOCAL (our domain) signature that is not related to the remote address. local dkim signature is useless for the spam checking and should not be assigned to the remote addr.. Actually in my configuration the DKIM signs the outgoing mail AFTER processing with SA. and it is okay. So the current logics for the DKIM isolation of the records and the auto whitelist outgoing mail is broken cause we can not sign outgoing mail with someone's key. That is why I proposed the algorythm where outgoing whitelist records should be processed separately ignoring the signedby field