figured out the perl_debugger ...

stepping through 'spamd' with perl_debugger, (actually a working copy i've named
'spamd_test.pl'), i follow into:

		sub daemonize {
		  # Pretty command line in ps
		  $0 = join (' ', $ORIG_ARG0, @ORIG_ARGV) unless would_log("dbg");
		
		  # be a nice daemon and chdir to the root so we don't block any
		  # unmount attempts
		  chdir '/' or die "spamd: cannot chdir to /: $!\n";
		
		  # Redirect in and out to the bit bucket
		  open STDIN,  "</dev/null" or die "spamd: cannot read from /dev/null: $!\n";
		  open STDOUT, ">/dev/null" or die "spamd: cannot write to /dev/null: $!\n";
		
		  # Remove the stderr logger
		  Mail::SpamAssassin::Logger::remove('stderr');
		
		  # Here we go...
2140:	  defined( my $pid = fork ) or die "spamd: cannot fork: $!\n";
2141:	  exit if $pid;
2142:	  setsid or die "spamd: cannot start new session: $!\n";
2143:	
2144:	  # Now we can redirect the errors, too.
2145:	  open STDERR, '>&STDOUT' or die "spamd: cannot duplicate stdout: $!\n";
2146:	
2147:	  dbg("spamd: successfully daemonized");
	}


i can bkpt *cleanly* TO the line 2140:

	  DB<1> c 2140
	main::daemonize(./spamd_test.pl:2140):
	2140:     defined( my $pid = fork ) or die "spamd: cannot fork: $!\n";
	  DB<2> 

and, if i then bkpt to line 2147:

	  DB<2> c 2147
	Debugged program terminated.  Use q to quit or R to restart,
	  use o inhibit_exit to avoid stopping after program termination,
	  h q, h R or h o to get additional info.  
	  DB<3> 


i get the crash:

	Sep 14 14:23:26 server crashdump[21963]: perl crashed
	Sep 14 14:23:27 server crashdump[21963]: crash report written to:
/Library/Logs/CrashReporter/perl.crash.log

as before ...

	Exception:  EXC_BAD_ACCESS (0x0001)
	Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000000
	
	Thread 0 Crashed:
	0   libSystem.B.dylib        	0x90131a80 bcmp + 224
	1   com.apple.CoreFoundation 	0x902c580c __CFInitialize + 420
	2   dyld                     	0x8fe15660
ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 672
	3   dyld                     	0x8fe0bbe8 
	...


if however, i bkpt to 2140, then try to single-step:

	  DB<1> c 2140
	main::daemonize(./spamd_test.pl:2140):
	2140:     defined( my $pid = fork ) or die "spamd: cannot fork: $!\n";
	  DB<2> s
	######### Forked, but do not know how to create a new TTY. #########
	  Since two debuggers fight for the same TTY, input is severely entangled.
	
	main::daemonize(./spamd_test.pl:2141):
	2141:     exit if $pid;
	  DB<2>   I know how to switch the output to a different window in xterms
	  and OS/2 consoles only.  For a manual switch, put the name of the created TTY
	  in $DB::fork_TTY, or define a function DB::get_fork_TTY() returning this.
	
	  On UNIX-like systems one can get the name of a TTY for the given window
	  by typing tty, and disconnect the shell from TTY by sleep 1000000.
	
	main::daemonize(./spamd_test.pl:2141):
	2141:     exit if $pid;
		{pid=21967}  DB<2> s
	Mail::SpamAssassin::Logger::CODE(0x18bf8f0)(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger.pm:319):
	319:      close_log();
	  DB<2> s
	Mail::SpamAssassin::Logger::close_log(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger.pm:313):
	313:      while (my ($name, $object) = each %{ $LOG_SA{method} }) {
	  DB<2> s
	Mail::SpamAssassin::Logger::close_log(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger.pm:313):
	313:      while (my ($name, $object) = each %{ $LOG_SA{method} }) {
	  DB<2> s
	Mail::SpamAssassin::Logger::close_log(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger.pm:314):
	314:        $object->close_log();
	  DB<2> s
	Mail::SpamAssassin::Logger::Syslog::close_log(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger/Syslog.pm:207):
	207:      my ($self) = @_;
	  DB<2> s
	Mail::SpamAssassin::Logger::Syslog::close_log(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger/Syslog.pm:209):
	209:      closelog();
	  DB<2> s
	Sys::Syslog::closelog(/usr/local/perl_libs/privlib/darwin-thread-multi-2level/Sys/Syslog.pm:133):
	133:        $facility = $ident = '';
	  DB<2> s
	Sys::Syslog::closelog(/usr/local/perl_libs/privlib/darwin-thread-multi-2level/Sys/Syslog.pm:134):
	134:        disconnect_log();
	  DB<2> s
	Sys::Syslog::disconnect_log(/usr/local/perl_libs/privlib/darwin-thread-multi-2level/Sys/Syslog.pm:628):
	628:        $connected = 0;
	  DB<2> s
	Sys::Syslog::disconnect_log(/usr/local/perl_libs/privlib/darwin-thread-multi-2level/Sys/Syslog.pm:629):
	629:        $syslog_send = undef;
	  DB<2> s
	Sys::Syslog::disconnect_log(/usr/local/perl_libs/privlib/darwin-thread-multi-2level/Sys/Syslog.pm:631):
	631:        if($current_proto eq 'native') {
	  DB<2> s
	Sys::Syslog::disconnect_log(/usr/local/perl_libs/privlib/darwin-thread-multi-2level/Sys/Syslog.pm:636):
	636:        return close SYSLOG;
	  DB<2> s
	Mail::SpamAssassin::Logger::close_log(/usr/local/perl_libs/sitelib/Mail/SpamAssassin/Logger.pm:313):
	313:      while (my ($name, $object) = each %{ $LOG_SA{method} }) {
	  DB<2> s
	Debugged program terminated.  Use q to quit or R to restart,
	  use o inhibit_exit to avoid stopping after program termination,
	  h q, h R or h o to get additional info.  
	  DB<2> q
	/usr/ports/temp  > main::daemonize(./spamd_test.pl:2142):
	2142:     setsid or die "spamd: cannot start new session: $!\n";
	{pid=21967}  DB<2> 


and, atm, not sure what to make of that ...  :-/

fwiw, this has been discussed a bit on the irc channel.  the problems started
occuring after upgrading a bunch of modules via the CPAN shell's "r" command.

if I ignore the module upgrades, it sounds like a perl bug -- perl shouldn't be
crashing on any of the standard statements that it seems to be crashing on.  but
if the problems only started after the modules were upgraded (and not perl being
upgraded), it sounds XS related somehow.

I've recommended trying to rollback the modules that were upgraded, or since
this is apparently a custom perl install, make a new clean version, don't
upgrade any of the standard modules, and install just enough extra modules to
get SA functional, then see what happens.

FWIW, I can run spamd from trunk on my OS X laptop w/ standard perl 5.8.6
install without issue.  I also tried upgrading HTML::Tagset, HTML::Parser,
Digest::SHA1, and Net::DNS to the version that he has on his machine, and things
continue to work fine.

BTW: on the irc channel I said there were 3 ways for perl to explode, but
there's really at least 5, though none of them are SA related:

- hardware problem
- perl binary issue
- perl threading issue
- perl XS module issue
- ulimits (memory limits definitely cause perl to have problems)
- compiler issue (can cause above binary/XS issues)

> problems started occuring after upgrading a bunch of modules via the CPAN
shell's "r" command.

that is true.

and, to complicate matters, when i'd upgraded, i did "cpan> r", then updated
those that needed updating were available ...

since then, i've rebuilt from scratch ... so my builds now have all the latest
modules.

i.e., i don't have the cpan module version history available ...

> if the problems only started after the modules were upgraded (and not perl being
upgraded), it sounds XS related somehow.
> I've recommended trying to rollback the modules that were upgraded, or since
this is apparently a custom perl install, make a new clean version, don't
upgrade any of the standard modules, and install just enough extra modules to
get SA functional, then see what happens.

fyi, i've been running perl 588 for ages on these boxes ... until the last
couple of days, quite successfully with SA-head, as well.

> FWIW, I can run spamd from trunk on my OS X laptop w/ standard perl 5.8.6
install without issue.  I also tried upgrading HTML::Tagset, HTML::Parser,
Digest::SHA1, and Net::DNS to the version that he has on his machine, and things
continue to work fine.

i've also reproduced the crash on two OSX-out-of-the-box boxes, with Apple's
perl 586.  To be fair, the details of the crash are different, and I have not
started tracing through a 'spamd' traversal with *that* perl version ...

> BTW: on the irc channel I said there were 3 ways for perl to explode, but
there's really at least 5, though none of them are SA related:

ok.

> - hardware problem

this is occurring/reproducible on several different boxes.

all, however are 'just' PowerPC (G#s & G$s ... no MacIntels)

> - perl binary issue
> - perl threading issue

again, this is the 'same' perl core that i've been using for awhile

> - perl XS module issue

possible, of course, ... 

> - ulimits (memory limits definitely cause perl to have problems)

fwiw, on all boxes:

	% ulimit -n
		1024

> - compiler issue (can cause above binary/XS issues)

dunno ...


in general, i understand that the inclination is to 'blame perl' ...

in this case, tho, i simply see no problems with perl on any box with any other app.

ONLY in the specific case of "spamd --daemonize", and then the problem traces to
daemon code ...

just fwiw, changing to:

% ulimit -n
  12288

has, alas, no effect ...

blamed on perl; stated as 'not related to SA'