btw I submitted a CLA, many years ago. Looks like bz has lost track of it :/

Nick, my apologies.  There is no automated update for bugzilla when a CLA is filed.  I have confirmed with the secretary your CLA is on file and updated your bugzilla account to reflect this.

Thanks for the patch and taking the time to mention the CLA issue.  -KAM

Hm, some large-scale sa-learn runs show that it is leaking tied filehandless like confetti too, because it ties to the storage whenever open_storages() is called, but unties only if one of the code paths through check_senders_reputation()->check_reputations()->check_reputation() happens to be invoked.

It is possible that the right fix is to change

  if (defined $storage) {
    $self->{checker} = $self->{default_storage};
  }

in check_reputation to

  if (!defined $storage) {
    $self->{checker} = $self->{default_storage};
  }

since it appears that without that we are currently only untying if we are using different storages per-user and per-system, but even this is error-prone and will leak if we take the wrong path down through these functions. It's easier just to cut the Gordian knot and untie from the storage whether or not $self->{checker} is set.

However, even if you fix this, it still leaks tx-reputation fhs like confetti and I have no idea why: ties and unties are precisely balanced, but the untie is not closing the fh.

If anyone can see what's wrong, I'd be happy to know... but right now I'm tempted to can this whole mess and go back to the AWL. At least the AWL works.

Created attachment 5579 [details]
txrep leak partial fix

I think I can commit this one.
It's not a complete fix for the file descriptor leak but at least is better than
what we have in-tree.

I committed a fix (maybe partial) for this in commit #1837876 in both trunk and 3.4.

Giovanni, do you think this is resolved or more patches needed?

From my tests this has been fixed with commit #1837876.

This is still present, I'm afraid. We don't seem to be actively blocking any more, but we are still leaking filehandles like a sieve. I filter three emails through it, and what do I see?

/proc/193902/fd:
total 0
lr-x------ 1 root root 64 Sep 18 20:16 0 -> /dev/null
l-wx------ 1 root root 64 Sep 18 20:16 1 -> /dev/null
l-wx------ 1 root root 64 Sep 18 20:16 2 -> /dev/null
lr-x------ 1 root root 64 Sep 18 20:16 3 -> /pkg/perl/5.26.2-14/usr/bin/spamd
lrwx------ 1 root root 64 Sep 18 20:16 4 -> 'socket:[78493696]'
lrwx------ 1 root root 64 Sep 18 20:16 5 -> 'socket:[78516226]'
lrwx------ 1 root root 64 Sep 18 20:16 6 -> 'socket:[78516228]'
lrwx------ 1 root root 64 Sep 18 20:16 7 -> 'socket:[78518296]'
lrwx------ 1 root root 64 Sep 18 20:16 8 -> 'socket:[78506775]'
lrwx------ 1 root root 64 Sep 18 20:16 9 -> /home/nix/.spamassassin/tx-reputation.mutex
lrwx------ 1 root root 64 Sep 18 20:16 10 -> /home/nix/.spamassassin/tx-reputation
lrwx------ 1 root root 64 Sep 18 20:16 11 -> /home/nix/.spamassassin/tx-reputation
lrwx------ 1 root root 64 Sep 18 20:16 12 -> /home/nix/.spamassassin/tx-reputation

The mutex is no longer being leaked, so it's not blocking any more, but a tied fh to the actual *database* is still leaked, so we'll soon enough run out of fhs.

Are we talking about a file descriptor leak or a blocking issue?

We used to see an fd leak and blocking (because one of the things being leaked was the fh to the mutex).

We're just seeing an fd leak now (one fh per message scanned), but despite the distinct symptoms it is really the same bug, I think.

Fixed in r1909836.