Bug 3729 - spamd is crashing hard...
Summary: spamd is crashing hard...
Status: RESOLVED WORKSFORME
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: spamc/spamd (show other bugs)
Version: SVN Trunk (Latest Devel Version)
Hardware: Macintosh Mac OS X
: P3 normal
Target Milestone: 3.0.1
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-08-25 12:08 UTC by jon stevens
Modified: 2004-10-10 00:37 UTC (History)
0 users


Attachment Type Modified Status Actions Submitter/CLA Status
this file causes the crash application/octet-stream None jon stevens [NoCLA]
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description jon stevens 2004-08-25 12:08:41 UTC 
Hi there, after upgrading SA to the latest SVN head (and not making any other
system changes), I'm getting spamd/perl to reliably crash within a few seconds
of starting up (probably as soon as it tries to process a message).

I'm not really sure how to debug this since the only thing that I'm seeing in
the logs is below.

I have tried removing all the autowhitelist and bayes files.

I have tried removing all the /Library/Perl/5.8.4/Mail/Spamassassin* files and
re-installing.

I have made sure that all the required CPAN modules (as documented in the
INSTALL file) are up to date.

I have confirmed that SA is not installed anywhere else on the system.


Summary of my perl5 (revision 5 version 8 subversion 4) configuration:
  Platform:
    osname=darwin, osvers=6.8, archname=darwin
    uname='darwin localhost 6.8 darwin kernel version 6.8: wed sep 10 15:20:55
pdt 2003; root:xnuxnu-344.49.obj~2release_ppc power macintosh powerpc '
    config_args='-de -Dprefix=/usr -Uloclibpth -Dlibpth=/usr/lib /sw/lib
-Dldflags=-L/usr/lib -L/sw/lib -Dlocincpth=/sw/include /sw/include/db4
-Dlibs=-ldb-4.1 -ldl -lm -lcrypt -lc'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-pipe -fno-common -DPERL_DARWIN -no-cpp-precomp
-fno-strict-aliasing -I/sw/include -I/sw/include/db4',
    optimize='-Os',
    cppflags='-no-cpp-precomp -pipe -fno-common -DPERL_DARWIN -no-cpp-precomp
-fno-strict-aliasing -I/sw/include -I/sw/include/db4'
    ccversion='', gccversion='3.1 20020420 (prerelease)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-L/usr/lib -L/sw/lib -flat_namespace'
    libpth=/usr/lib /sw/lib
    libs=-ldb-4.1 -ldl -lm -lcrypt -lc
    perllibs=-ldb-4.1 -ldl -lm -lcrypt -lc
    libc=/usr/lib/libc.dylib, so=dylib, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-L/usr/lib -L/sw/lib -flat_namespace -bundle
-undefined suppress'


Characteristics of this binary (from libperl): 
  Compile-time options: USE_LARGE_FILES
  Built under darwin
  Compiled at Jun  7 2004 12:29:39
  %ENV:
    PERL5LIB="/sw/lib/perl5"
  @INC:
    /sw/lib/perl5/darwin
    /sw/lib/perl5
    /System/Library/Perl/5.8.4/darwin
    /System/Library/Perl/5.8.4
    /Library/Perl/5.8.4/darwin
    /Library/Perl/5.8.4
    /Library/Perl
    /Network/Library/Perl/5.8.4/darwin
    /Network/Library/Perl/5.8.4
    /Network/Library/Perl



Starting spamd services
trying to connect to syslog/unix...
no error connecting to syslog/unix
logging enabled:
        facility: mail
        socket:   unix
        output:   syslog
creating INET socket:
        Listen: 128
        LocalAddr: 127.0.0.1
        LocalPort: 783
        Proto: 6
        ReuseAddr: 1
        Type: 1
debug: SpamAssassin version 3.0.0-rc1-r36452
debug: Score set 0 chosen.
debug: Storable module v2.13 found


******************

Date/Time:  2004-08-25 12:06:04 -0700
OS Version: 10.2.8 (Build 6R73)
Host:       takahe.whichever.com

Command:    perl
PID:        4006

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_INVALID_ADDRESS (0x0001) at 0xbff7fff0

Thread 0 Crashed:
 #0   0x90004258 in free_list_add_ptr
 #1   0x90003f28 in szone_free
 #2   0x0002ddc0 in Perl_sv_clear
 #3   0x0002e100 in Perl_sv_free
 #4   0x00012154 in Perl_peep
 #5   0x00011f90 in Perl_peep
 #6   0x00011f90 in Perl_peep
...
(repeats like #6 for a bazillion more times)
Comment 1 Daniel Quinlan 2004-08-25 12:24:55 UTC 
If Perl is outright crashing, then it's either a MacOS bug, a Perl bug,
or a MacOS Perl bug.  

I did a bugzilla search for MacOS bugs mentioning the words "crash", "core",
"die", or "dies" and 8 out of 9 of them were reported by you.  Something is
probably wrong with your machine or your installation of Perl.
Comment 2 Justin Mason 2004-08-25 12:27:01 UTC 
what does an strace produce (or the equivalent for OSX)?
It looks a lot like a crash in Mail::SA::copy_config() caused by a bug in Storable.
Comment 3 Daniel Quinlan 2004-08-25 12:28:57 UTC 
Subject: Re:  spamd is crashing hard...

In all fairness, you've reported at least 2/3 of the MacOS bugs, but I'm
still pretty certain this is a MacOS/Perl problem.
Comment 4 jon stevens 2004-08-25 12:31:12 UTC 
The software has been working fine for months now. I upgrade SA to latest SVN
HEAD and it stops working. Can't you possibly see that it might be a problem in
SA? If you go through my bugs, you will see that several of them turned out to
be obscure problems in SA.
Comment 5 jon stevens 2004-08-25 12:48:36 UTC 
http://whichever.com/kdump.out.gz
Comment 6 jon stevens 2004-08-25 12:51:39 UTC 
I will also note that my installation of Perl is perfect. I spent a bunch of
time removing all the old OSX perl installation and have a very clean install at
this point. The *only* thing that could possibly be messed up is that I'm using
db41 instead of db42. However, since it was working just fine for months
previously, I don't see why that should now be a problem other than something SA
is causing. I will work on upgrading to db42/perl585 though just to make sure.
Comment 7 Michael Parker 2004-08-25 12:58:55 UTC 
Subject: Re:  spamd is crashing hard...

What version did you upgrade from?  Might help track down any changes
that have been made.

BTW: I seem to remember serious problems with Berkeley DB 4.1.x on
OSX.

Changes where made in 3.0 that seemed to tickle previously un-tickled
portions of the BDB code.

Michael
Comment 8 jon stevens 2004-08-25 13:11:34 UTC 
Unfortunately, I'm not sure which SVN head I upgraded from. =(

I'm working on compiling/installing db42 and perl585. Will report back if that
works.
Comment 9 Justin Mason 2004-08-25 13:13:00 UTC 
interestingly, it seems to make it a lot further in the kdump.  last debug
line output is

 18074 perl     GIO   fd 2 wrote 54 bytes
       "logmsg: debug: running uri tests; score so far=-2.623
       "

that's quite a bit further than the previous trace, which died before spamd had
even daemonized.  I'd prefer to see one that failed in the same place ;)

however... that would be consistent with spamd coredumping due to memory
corruption caused by a bad copy of the Conf object using Storable, so I'm
sticking with that theory.

Also, Storable was added as a required module relatively recently in 3.0.0
development.

Jon, what spamd flags are you using?
Comment 10 Theo Van Dinter 2004-08-25 13:14:06 UTC 
>The software has been working fine for months now. I upgrade SA to latest SVN
>HEAD and it stops working. Can't you possibly see that it might be a problem in SA?

Just to note: if perl is actively dying, such as sig 11, throwing exceptions, etc, then it's not a problem 
with SpamAssassin itself since we don't do any compiled code accesses inside our code (aka: no XS, 
etc.)  There's no way for a perl program that is pure perl (such as SpamAssassin) to cause those types 
of errors without a bug in the perl binary itself.

If there's an issue in perl itself or with other compiled/XS modules on Mac OS X, there's really not much 
we can do about it.  That's just one of the problems with code reuse libraries/modules -- you have to 
rely that they don't have bugs that impact your code.
Comment 11 jon stevens 2004-08-25 13:30:18 UTC 
/usr/bin/spamd -D -m 1 -d -r /var/run/spamd.pid

I told ktrace to output for all children so that it would catch the most amount
of information to help with debugging...

     -d      Descendants; perform the operation for all current children of
             the designated processes.
     -i      Inherit; pass the trace flags to all future children of the des-
             ignated processes.

Once again, I have been using SVN HEAD of SA 3.x for a while now and has been
working just fine. No SA 2.x is on my system anymore.

jon
Comment 12 Daniel Quinlan 2004-08-25 13:41:25 UTC 
Subject: Re:  spamd is crashing hard...

> Once again, I have been using SVN HEAD of SA 3.x for a while now and has been
> working just fine. No SA 2.x is on my system anymore.

It might help to know which version of SVN broke it for you.

Binary search on SVN versions is probably the more reliable way with
some reasonably conservative guess as to the original version that was
working.
Comment 13 Justin Mason 2004-08-25 15:58:17 UTC 
btw Jon -- I have a feeling that upgrade perl from 5.8.4, or the DB libraries,
will not help here; I think it's a Storable bug, poss platform-specific.   I
think the svn-version binary search is the most effective next step.
Comment 14 jon stevens 2004-08-26 01:30:55 UTC 
Created attachment 2291 [details]
this file causes the crash

I reverted to this release:

http://spamassassin.apache.org/released/Mail-SpamAssassin-3.0.0-rc1.tar.gz

I then remembered that I have a ton of .cf rules from RDJ.

So, I went through the ones that had changed over the last couple of days and
tested starting SA without each file. I then narrowed it down to this one file
(blacklist-uri.cf) that is causing the problems. I also noticed that it causes
the crash during the test spam message and SA never even gets a chance to spawn
a child process.

I just tried with svn head and no blacklist-uri file and that is definitely the
problem.

I suspect that if you diff this file with the previous version of it (which I
don't have a copy of and can't seem to locate on their server!), it should be
possible to narrow down the specific change that caused the problem.

Given the trust nature of RDJ, it seems pretty dangerous to allow a rule to
take down someone's SA. Since it is the virtual machine that is crashing, I'm
not really sure how to properly handle this one. Just something to think about.
Comment 15 Daniel Quinlan 2004-08-26 14:03:04 UTC 
Subject: Re:  spamd is crashing hard...

> I suspect that if you diff this file with the previous version of it (which I
> don't have a copy of and can't seem to locate on their server!), it should be
> possible to narrow down the specific change that caused the problem.

Can you narrow it down to one rule or a set of rules in that file?

Something like:

a. divide file in half, see which half fails, keep dividing until you
   find the smallest part that still fails

b. if that doesn't work, try eliminating one rule at a time until you
   get something that doesn't fail, add back that last rule that caused
   it to fail, skip to the next rule... and keep trying to remove rules
   until you can't remove any rule without causing it to NOT crash

Daniel
Comment 16 Matt Kettler 2004-08-27 13:17:36 UTC 
Sidenote: blacklist-uri is a rather large rulefile.. it's 740k+ of high-density
regexes.

How much ram does your system have? You may causing your system to run out of
memory causing the crash.

Beware of using large rulesets like blacklist-uri unless you have a LOT of free
ram laying around (i.e.: I'd suggest having at least 200mb of free physical
memory before starting spamd with a large config)

It's also a bit redundant to use blacklist-uri.cf with SA 3.0, since
blacklist-uri.cf has been imported to ws.surbl.org, and is queried via the URI
DNSBL feature that SA 3.0 has built in. 

The only reason you should even consider using this .cf file is if you don't
want to use network tests and have lots of spare ram.

There are acutaly quite a few RDJ-type rulesets that are already built into SA
3.0 (ie: antidrug).. You might want to consider trimming down.
Comment 17 Justin Mason 2004-08-27 15:29:17 UTC 
'How much ram does your system have? You may causing your system to run out of
memory causing the crash.'

if not running out of memory directly, at least running into process ulimits. 
that is a possibility alright...
Comment 18 Daniel Quinlan 2004-08-27 17:35:55 UTC 
bugs for 3.0.1 milestone
Comment 19 Theo Van Dinter 2004-10-01 12:32:32 UTC 
lowering the priority on this one
Comment 20 Theo Van Dinter 2004-10-10 08:37:26 UTC 
since the problem seems to be using third-party rulesets, and there was no more
information provided (like the rule(s) that cause the problem), there's nothing
more we can do.  closing as WFM for now.