Bug 7518 - spamc bounces e-mail with more than 16 receipients
Summary: spamc bounces e-mail with more than 16 receipients
Status: RESOLVED WONTFIX
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: spamc/spamd (show other bugs)
Version: 3.4.0
Hardware: Other Linux
: P2 normal
Target Milestone: 3.4.2
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-06 10:38 UTC by Dirk
Modified: 2018-08-28 23:49 UTC (History)
5 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk 2017-12-06 10:38:56 UTC
Hi,

on monday one of our users reported a bounced outgoing email. After analysing the maillog we found 17 bounces of the same postfix email-queue-id.

This is the error-text:

Dec  4 17:24:32 nwomx postfix/pipe[40200]: 5973C906554: to=<foo@bar.de>, relay=spamchk, delay=0.18, delays=0.12/0.01/0/0.05, dsn=5.3.5, status=bounced (local configuration error. Command output: Exceeded max number of arguments (24) in /etc/mail/spamassassin/spamc.conf 

spamc.conf:
-s 268435456

After some tests we can reproduce this behavoiur when sending an email to more than 16 receipients.

Any ideas?

Regards

Dirk


Environment:
============
OS: CentOS Linux release 7.4.1708 (Core)
Spamassassin: 3.4.0-2.el7

Postfix-Integration:
spamchk   unix  -       n       n       -       -       pipe
  flags=Rq user=spam argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
Comment 1 AXB 2017-12-06 10:42:35 UTC
SpamAssassin does not bounce mail. I can only tag and or log.
Please, do not abuse the bug tracker as a support tool.
Use SA user mailing list for such questions.
Comment 2 Dirk 2017-12-06 10:49:48 UTC
As you can see in the log snipplet there is an error coming from spamc which causes postfix to bounce the email.

The error states a configuration error in spamc.conf, which is not true as you can see also in my message.

So i think i am not trying to use this tracker as a support tool!

SPAMC produces this error when checking emails with more than 16 receipients (double-checked with test-emails).

So do you still think the is not a bug in spamc?
Comment 3 Kevin A. McGrail 2017-12-06 10:54:38 UTC
(In reply to Dirk from comment #2)
> As you can see in the log snipplet there is an error coming from spamc which
> causes postfix to bounce the email.
> 
> The error states a configuration error in spamc.conf, which is not true as
> you can see also in my message.
> 
> So i think i am not trying to use this tracker as a support tool!
> 
> SPAMC produces this error when checking emails with more than 16 receipients
> (double-checked with test-emails).
> 
> So do you still think the is not a bug in spamc?

Configuration issue NOT a bug. This is not a support form.  Subscribe to users@spamassassin.apache.org and discuss there.  Do not reopen bugs.
KAM
Comment 4 Benny Pedersen 2017-12-06 11:51:52 UTC
http://postfix.1071664.n5.nabble.com/Split-multiple-recipient-mail-td48458.html

its multirecipient limit

<transport>_destination_recipient = 1 

mta stage is no limit, but content filter transport should hae this in main.cf

sorry for write after its declared invalid :=)
Comment 5 Dirk 2017-12-06 13:34:26 UTC
(In reply to Benny Pedersen from comment #4)
> http://postfix.1071664.n5.nabble.com/Split-multiple-recipient-mail-td48458.
> html
> 
> its multirecipient limit
> 
> <transport>_destination_recipient = 1 
> 
> mta stage is no limit, but content filter transport should hae this in
> main.cf
> 
> sorry for write after its declared invalid :=)

Thanks, that helps as a workaround for this behaviour. Setting the limit to 15 works fine now.

Dirk
Comment 6 Dirk 2017-12-08 08:11:56 UTC
(In reply to Benny Pedersen from comment #4)
> http://postfix.1071664.n5.nabble.com/Split-multiple-recipient-mail-td48458.
> html
> 
> its multirecipient limit
> 
> <transport>_destination_recipient = 1 
> 
> mta stage is no limit, but content filter transport should hae this in
> main.cf
> 
> sorry for write after its declared invalid :=)

A last post to maybe help understanding why this is truely a bug in spamc:

First of all, i am not only an administrator, i am a programner too! I know what i am talking about and i hate quick answers "your are stupid, shut up and go home" without further reading and just trying to understand if the user could be right! I can fix the code by myself, but i am not responsible!

So please read the text below, think about it and than do what ever you want.

After analysing the error in detail (the args-exceeded comes from spamc) the problem relies in the argument-parser of spamc.

The switch "-e" has only ONE argument up to EOL (as stated in the code and manpage). The code of SPAMC counts ALL arguments until EOL, so all arguments passed to sendmail (in my example) will increase the arg-counter in spamc.

This results in an ARGC-overflow with emails with more than 16 recipients (spamc counts "-s VALUE" in the configfile and "-e", "SENDMAIL", "-oi", "-t", "SENDER" and "ALL recipients"! This results in 25 counted arguments with 17 recipients which causes spamc to abort!

Reducing the number of arguments by multiplying the email within the MTA is definitly just a workaround! This leads into scanning the same email more than once, leading to a much higher load for the server.

If you further think this is not a bug, you should mention this in the manpage, because the text

Note that this must be the LAST command line option, as everything after the -e is taken as arguments to the command (it's like rxvt or xterm)."

says that "-e" has only ONE argument.

Regards

Dirk
Comment 7 Kevin A. McGrail 2017-12-08 08:52:32 UTC
Fair enough that you might have a bug but spamc does NOT bounce emails hence the confusion.

However, the call from postfix to spamc is an integration issue it seems.

Can you demonstrate this issue from the command line calling spamc without postfix involved at all?
Comment 8 Dirk 2017-12-08 09:09:40 UTC
Hi,

thanks for the reply and sorry for the "wrong" subject!

To proof the behaviour:

This works fine:

[root@nwomx local]# spamc -e /usr/sbin/sendmailx -oi -f dirk.traenapp@nwowhv.de test1@gmail.com test2@gmail.com test3@gmail.com test4@gmail.com test5@gmail.com test6@gmail.com test7@gmail.com test8@gmail.com test9@gmail.com test10@gmail.com test11@gmail.com test12@gmail.com test13@gmail.com test14@gmail.com test15@gmail.com test16@gmail.com   < /tmp/test.msg

Here it crashes:

[root@nwomx local]# spamc -e /usr/sbin/sendmail -oi -f dirk.traenapp@nwowhv.de test1@gmail.com test2@gmail.com test3@gmail.com test4@gmail.com test5@gmail.com test6@gmail.com test7@gmail.com test8@gmail.com test9@gmail.com test10@gmail.com test11@gmail.com test12@gmail.com test13@gmail.com test14@gmail.com test15@gmail.com test16@gmail.com test17@gmail.com  < /tmp/test.msg

As you can see this is exactly the same but just one more recipient.

For me this is not an integration problem! Sure, i can use a wrapperscript around the spamc-call, but this is just overhead and a lot of people use this kind of call directly out of postfix.

Regards

Dirk
Comment 9 Dirk 2017-12-08 09:11:11 UTC
Sorry, missed the error line :-D

[root@nwomx local]# spamc -e /usr/sbin/sendmail -oi -f dirk.traenapp@nwowhv.de test1@gmail.com test2@gmail.com test3@gmail.com test4@gmail.com test5@gmail.com test6@gmail.com test7@gmail.com test8@gmail.com test9@gmail.com test10@gmail.com test11@gmail.com test12@gmail.com test13@gmail.com test14@gmail.com test15@gmail.com test16@gmail.com test17@gmail.com  < /tmp/test.msg 
Exceeded max number of arguments (24) in /etc/mail/spamassassin/spamc.conf
Comment 10 Kevin A. McGrail 2017-12-08 09:26:26 UTC
(In reply to Dirk from comment #9)
> Sorry, missed the error line :-D
> 
> [root@nwomx local]# spamc -e /usr/sbin/sendmail -oi -f
> dirk.traenapp@nwowhv.de test1@gmail.com test2@gmail.com test3@gmail.com
> test4@gmail.com test5@gmail.com test6@gmail.com test7@gmail.com
> test8@gmail.com test9@gmail.com test10@gmail.com test11@gmail.com
> test12@gmail.com test13@gmail.com test14@gmail.com test15@gmail.com
> test16@gmail.com test17@gmail.com  < /tmp/test.msg 
> Exceeded max number of arguments (24) in /etc/mail/spamassassin/spamc.conf

There is a limit of 24 arguments set for spamc in 
spamc.h.in:#define COMBO_ARGV_SIZE 24

The program is bombing out because of the number of arguments

You are using the -e option which has the statement "Note that this must be the LAST command line option"

So yes, you are passing invalid arguments to spamc.  

I'm not seeing a bug here.

Is there a way to put all those email addresses in quotes or something so sendmail gets them as one arg?

Limit your max recipients to 21 or whatever number you bomb at so you only rescan for 2x for 42 recips.

Change your spamc.h to a higher number, test and run that. 

I'm not familiar with postfix or how you are calling spamc

I'm convinced you have problem just not that you have a bug.  Spamc is failing because it's getting too many arguments.
Comment 11 Dirk 2017-12-08 11:56:22 UTC
I know about the DEFINE, thanks, but i do not want to build my own packages.

Quoting the call from postfix does not work, so i will further use the workaround by limiting within postfix.
Comment 12 Kevin A. McGrail 2017-12-08 12:00:17 UTC
I would recommend looking at other ways to call spamc from postfix.  Your way seems odd.  Why does it need to know the sender and all the recipient emails again, for example?  

It's like instead of using a filter you are resending the email.
Comment 13 RW 2017-12-08 12:55:26 UTC
IMO running it like this is bad practice because you're exposing all the recipients to the settings and training of one of them.  

My understanding  is that people usually either process once per user or use a special default user when there are multiple recipients.
Comment 14 Bill Cole 2017-12-08 20:20:45 UTC
(In reply to Kevin A. McGrail from comment #12)
> I would recommend looking at other ways to call spamc from postfix.  Your
> way seems odd.  Why does it need to know the sender and all the recipient
> emails again, for example?  

This is (unfortunately) a method documented on the SpamAssassin Wiki: https://wiki.apache.org/spamassassin/IntegratedSpamdInPostfix

> It's like instead of using a filter you are resending the email.

This is a documented and supported Postfix mechanism for after-queue filtering. Messages being delivered locally are piped into the "content_filter" command which typically also gets a trailing set of arguments like '-G -oi -f ${sender} ${recipient}' which are then handed off to sendmail for local re-injection & delivery.

Since the spamc '-e' option is documented as accepting an unspecified plural number of arguments to pass to the output command which is in fact limited by an undocumented variable in an Autoconf input file, this seems like a documentation bug, a misleading error message (since you cannot change COMBO_ARGV_SIZE without reconfiguring the source and rebuilding,) and maybe an ill-considered design choice. 

The most obvious "fix" would be to note the COMBO_ARGV_SIZE limit in the spamc man page and on the aforementioned Wiki page.
Comment 15 Kevin A. McGrail 2017-12-08 20:36:14 UTC
(In reply to Bill Cole from comment #14)
> The most obvious "fix" would be to note the COMBO_ARGV_SIZE limit in the
> spamc man page and on the aforementioned Wiki page.

Makes sense to me.  And I agree this is a "sorta" bug.  the bad title led us astray but we've moved on from that.

Thought: Why can't that ARGV_SIZE be like 256 to be a more sane limit. With modern kernels and bash, is there really a limit?  

Then we can just mention the limit and update the docs including a caveat that it might exceed limits for the local os.

And we should update the wiki for the postfix setting that if you set it to 20 will run the filter for 20 of the users at a time.  

Obviously you can set it to 1 and it will always work but be hella inefficient as he points out above :-)

KAM
Comment 16 RW 2017-12-09 13:30:26 UTC
(In reply to Kevin A. McGrail from comment #15)

> Thought: Why can't that ARGV_SIZE be like 256 to be a more sane limit. 

The trouble with this is that it supports a bad practice. If postfix can't do it directly then spamc needs a default user parameter that's used when multiple recipients are passed (as in spamass-milter).
Comment 17 Kevin A. McGrail 2017-12-10 17:37:29 UTC
(In reply to RW from comment #16)
> (In reply to Kevin A. McGrail from comment #15)
> 
> > Thought: Why can't that ARGV_SIZE be like 256 to be a more sane limit. 
> 
> The trouble with this is that it supports a bad practice. If postfix can't
> do it directly then spamc needs a default user parameter that's used when
> multiple recipients are passed (as in spamass-milter).

Bad practice aside (since it's listed on our wiki), do you now if it will technically work?
Comment 18 Kevin A. McGrail 2018-08-28 23:49:30 UTC
Workarounds as noted to change the define or limit the postfix number to 15