Bug 7357 - spamc memory leak
Summary: spamc memory leak
Status: RESOLVED WORKSFORME
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: spamc/spamd (show other bugs)
Version: 3.4.1
Hardware: PC Linux
: P2 normal
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-10-11 19:36 UTC by marc
Modified: 2018-08-25 23:47 UTC (History)
3 users (show)


Attachment Type Modified Status Actions Submitter/CLA Status
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description marc 2016-10-11 19:36:32 UTC 
I run a low-volume spamd and exim on Debian.

Lately, the spamd children keep growing and growing, until one gets killed by the kernel oom-killer.  When first started, each child starts at about 100MB.  By they time they get killed, a child can grow to about 1.5GB.  The trouble is when it gets really fat, it has trouble forking dcc and pyzor.

I don't run any special custom rules.  Mostly stock stuff with a few plugins enabled (Pyzor, Razor, DCC, TxRep, Bayes, ASN, RelayCountry).  If I disable those plugins, there's still a (slower) memory leak.  Rules are pre-compiled.  A simple message, repeated enough times will eventually grow out of control.

Spamd gets started by the standard Debian scripts with the following options:
/usr/sbin/spamd --nouser-config --username=debian-spamd --groupname=debian-spamd --max-children 5 --helper-home-dir=/var/lib/spamassassin/ --listen /run/spamd.socket -d --pidfile=/var/run/spamd.pid

My 'fix' is to restart spamd with a cron job.  But that's just a band-aid solution.

I can try and help further diagnose the problem, but I wouldn't even know where to start.

Any help is appreciated.

Thanks,

Marc


debian-spamd@mail:~$ while true; do cat /tmp/test1.eml | spamc -y -U /run/spamd.socket ; echo ; ps xv | grep spamd; echo ; done
ALL_TRUSTED,BAYES_00,MISSING_HEADERS,PYZOR_CHECK
 4408 ?        R      0:04      0     0 234380 113324  5.5 spamd child
 4409 ?        S      0:00      0     0 223512 99632  4.8 spamd child
 4424 pts/5    R+     0:00      0   200 15215  1968  0.0 grep spamd

ALL_TRUSTED,BAYES_00,MISSING_HEADERS,PYZOR_CHECK
 4408 ?        S      0:04      0     0 234832 113788  5.5 spamd child
 4409 ?        R      0:04      0     0 234512 113404  5.5 spamd child
 4431 pts/5    S+     0:00      0   200 15215  1904  0.0 grep spamd

ALL_TRUSTED,BAYES_00,MISSING_HEADERS,PYZOR_CHECK
 4408 ?        R      0:09      0     0 241564 120500  5.8 spamd child
 4409 ?        S      0:04      0     0 234816 113692  5.5 spamd child
 4443 pts/5    S+     0:00      0   200 15215  1904  0.0 grep spamd

ALL_TRUSTED,BAYES_00,MISSING_HEADERS,PYZOR_CHECK
 4408 ?        S      0:09      0     0 241844 120680  5.8 spamd child
 4409 ?        R      0:09      0     0 241556 120532  5.8 spamd child
 4450 pts/5    S+     0:00      0   200 15215  1884  0.0 grep spamd

^C



This is the message:

Return-path: <marc@mclemente.net>
Envelope-to: marc@mclemente.net
Delivery-date: Sun, 09 Oct 2016 07:29:22 -0500
Received: from localhost ([::1] helo=mclemente.net)
	by mail.mclemente.net with smtp (Exim 4.87)
	(envelope-from <marc@mclemente.net>)
	id 1btDDu-0002gS-CC
	for marc@mclemente.net; Sun, 09 Oct 2016 07:29:21 -0500
Subject: test1
Message-Id: <E1btDDu-0002gS-CC@mail.mclemente.net>
From: marc@mclemente.net
Date: Sun, 09 Oct 2016 07:29:04 -0500

test1
Comment 1 RW 2016-10-13 13:33:01 UTC 
(In reply to marc from comment #0)

> If I disable those plugins, there's still a (slower) memory leak.

Did you fully disable the plugins by commenting-out the loadplugin line? 

> My 'fix' is to restart spamd with a cron job.  But that's just a band-aid
> solution.

By default spamc processes self-destruct after 200 scans. A better workaround would be to set --max-conn-per-child to something lower - I'd start with 10. It doesn't make all that much difference what you set it to as most initialization is done in the parent process. If necessary it could be set to 1, which would (hopefully) eliminate the problem.
Comment 2 RW 2016-10-13 14:19:54 UTC 
(In reply to RW from comment #1)

> By default spamc processes self-destruct after 200 scans. 

That should have been spamd child processes.
Comment 3 marc 2016-10-13 19:36:22 UTC 
The subject should have read spamd not spamc.

(In reply to RW from comment #1)
> (In reply to marc from comment #0)
> 
> > If I disable those plugins, there's still a (slower) memory leak.
> 
> Did you fully disable the plugins by commenting-out the loadplugin line? 

Yes.  Plugins were disabled by commenting out the respective loadplugin lines.

> > My 'fix' is to restart spamd with a cron job.  But that's just a band-aid
> > solution.
> 
> By default spamc processes self-destruct after 200 scans. A better
> workaround would be to set --max-conn-per-child to something lower - I'd
> start with 10. It doesn't make all that much difference what you set it to
> as most initialization is done in the parent process. If necessary it could
> be set to 1, which would (hopefully) eliminate the problem.

I am not sure that on my implementation the spamd children self-destruct after 200 messages.  But after reading your comment, I set --max-conn-per-child to 50 messages.  This gets the spamd children to about 500MB before they get killed.

While this is good to keep the server running without interruption, it is still a band-aid solution.  Ideally, the source(s) of the memory leaks should be found and fixed.

Marc
Comment 4 Kevin A. McGrail 2018-08-25 23:47:02 UTC 
Afraid you'll need to discuss this on users at or look for issues on your implementation.  We need more information to help you