transparency [was: database copying and compacting]

[David Relson]

On Tue, 9 Nov 2004 11:29:40 -0000 (GMT)
Robin Bowes wrote:

> On Tue, November 9, 2004 8:59, Matthias Andree said:
> > Clint Adams <schizo at debian.org> writes:
> >
> >
> >>> echo set_flags DB_DB_LOG_AUTOREMOVE >>~/.bogofilter/DB_CONFIG echo
> >>> set_lg_max 262144 >>~/.bogofilter/DB_CONFIG echo set_flags
> >>> DB_TXN_NOT_DURABLE >>~/.bogofilter/DB_CONFIG
> >>>
> >>
> >> Are there plans to wrap these in bogofilter config parameters so
> >users> won't have to muck around with the db env?
> >
> > Not at this time, why duplicate such code or make efforts to hide
> > that we're using BerkeleyDB? I have added recovery to bogoutil
> > because of the locking requirements, not as a fabric softener.
> 
> I think serious thought needs to be given to how transparent the use
> of Berkeley DB is. As a user of bogofilter I don't want to be bothered
> with the nitty gritty of configuring the underlying database tool - it
> should just work; if this is not the case then the question needs to
> be asked: is this the right tool for the job?
> 
> My suspicion is that it *is* the right tool but that it needs better
> integration with the bogofilter toolset before it is ready for
> prime-time.

Clint, Robin, Matthias,

I tend to agree.  I'd like to see BerkeleyDB use be as transparent as
possible.  Needing special copy and compact scripts is not ideal, nor is
knowing the nitty gritty details of BerkeleyDB locking or maintaining a
second config file.

However transactions _do_ provide value, so some knowledge of BerkeleyDB
details is necessary.  I'm wondering what can be done to make it easier.

Here's one idea:

At present, a number of DB_CONFIG options have been mentioned as useful.
A quick scan of README.db and recent messages shows these four:

    set_lk_max_objects  32768
    set_lk_max_locks    32768
    set_flags DB_LOG_AUTOREMOVE
    set_flags DB_TXN_NOT_DURABLE

A scan of /usr/include/db4.h shows that these can all be set via the
API.  As an alternative to the separate DB_CONFIG file, why not expand
bogofilter.cf to include these?  Possibly:

DB_MAX_LOCKS=32768        # Maximum number of locks.
DB_MAX_OBJECTS=32768      # Maximum number of locked objects.
DB_LOG_AUTOREMOVE=Yes/No  # Automatically remove log files.
DB_TXN_DURABLE=Yes/No     # Log changes

Matthias,

What other flags might be useful?

Clint and Robin,

Would this be helpful?  

David