Subject: Re:   New: auto-upgrade Berkeley DB files

On Tue, Jun 21, 2005 at 12:23:01PM -0700, bugzilla-daemon@bugzilla.spamassassin.org wrote:
> if that gets in, we could possibly use BerkeleyDB (if installed) to auto-upgrade
> dbs.

Then we should just use BerkeleyDB for all access.

*** Bug 4974 has been marked as a duplicate of this bug. ***

re the availabilty of BerkeleyDB -- Module::CoreList bears bad news:

: jm 202...; corelist BerkeleyDB
BerkeleyDB was not in CORE (or so I think)

compare with

: jm 203...; corelist DB_File
DB_File was first released with perl 5

for distro packages -- http://www.szabgab.com/distributions/letters/B.html says
that BerkeleyDB is packaged in Debian, FreeBSD, SuSE, Gentoo and ActivePerl.  I
think it's wrong though since my Ubuntu desktop says:

: jm 915...; apt-cache search berkeleydb
libberkeleydb-perl - use Berkeley DB 4 databases from Perl

(probably a universe package).

anyway, it's availability isn't great.  We *could* use it when available, using
its spiffy features, and fallback to DB_File where not, I suppose.

(In reply to comment #0)
> if that gets in, we could possibly use BerkeleyDB (if installed) to
auto-upgrade dbs.

in the bug, the developer says "I already have this in my development copy, must
do a release soon."  that was in 0.26, and it's now at 0.33, but I can't see any
upgrade() methods. hmm.

(In reply to comment #3)
> anyway, it's availability isn't great.  We *could* use it when available, using
> its spiffy features, and fallback to DB_File where not, I suppose.

fwiw, I was thinking of making a new BerkeleyDB BayesStore module, as opposed to making DBM more 
complicated than it already is.

(In reply to comment #5)
> fwiw, I was thinking of making a new BerkeleyDB BayesStore module, as opposed
to making DBM more complicated than it already is.

ok, good point, we have enough complexity there alright. ;)
that worked fine for Mail::SpamAssassin::BayesStore::SDBM....

> I was thinking of making a new BerkeleyDB BayesStore module,
> as opposed to making DBM more complicated than it already is.

That would be nice (although I've given up the use of bdb for
Bayes and AWL in favour of SQL a long time ago - it was causing
too much maintenance trouble and slowness when database grow
very large).

I'm using bdb databases in amavisd-new since v2.0, July 2004,
and doing it exclusively through a BerkeleyDB module with explicit
subroutine calls, avoiding a hash tie in order to have full control
on options and results of operations. Four databases are used:
three hashes and one queue (SNMP-like stats counters, process status
for each child process, a cache of recent results, and a queue for
purging a cache). Fine-grained locking is used at a record level,
using "Berkeley DB Concurrent Data Store" (cursors) for the purpose,
so database is never locked globally. The bdb V3.1 or better is
required, V4 is recommended.

I'm quite happy with how BerkeleyDB has performed its task, and its
keeping up with new versions of libdb. There were some quirks in
the underlying libdb, but recent versions (4.3, 4.4, 4.5, 4.6)
work very well for my purposes, even under heavy concurrent use.

There were hardly any complaints about BerkeleyDB not being part
of the Perl core modules. SpamAssassin and amavisd-new use plenty
of non-core modules, so BerkeleyDB is just one more.

(In reply to comment #5)
> I was thinking of making a new BerkeleyDB BayesStore module, as opposed
> to making DBM more complicated than it already is.

Btw, this is now covered by Bug 6046
(still unfinished, but promising)