database environment [was: bogofilter 0.93.4 - new current release]

[David Relson]

On Sun, 9 Jan 2005 10:16:54 -0800
Chris Wilkes wrote:

> On Sun, Jan 09, 2005 at 12:49:37PM -0500, David Relson wrote:
> > Hi Chris,
> > 
> > Sorry to hear you're encountering difficulties.  Let's see if we can
> > figure what's going on...
> > 
> > On Sun, 9 Jan 2005 09:30:02 -0800
> > Chris Wilkes wrote:
> > 
> > > On Sun, Jan 09, 2005 at 10:38:57AM -0500, David Relson wrote:
> > > > 
> > > > The 0.93.4 includes additional tweaks for supporting Berkeley DB
> > > > transactions, new options for ./configure for russian language support
> > > > (specifically "--enable-russian" and "--default-charset=charsetname"),
> > > > and a common "long options" file shared by bogofilter, bogolexer, and
> > > > bogoutil.  The usual number of small changes is also included.  Read
> > > > below for further details.
> > > 
> > > I suffered from the same lock table errors reported by others in 0.93.3
> > > when trying to do a bf_compact.  I updated to 0.93.4 and tried again:
> > > 
> > > $ bf_compact $BOGOFILTER_DIR wordlist.db
> > >   mv: cannot overwrite directory `/home/vmail/maildirs/cwilkes/.bogofilter/.old/.bogofilter'
> > > $ rmdir /home/vmail/maildirs/cwilkes/.bogofilter/.old/.bogofilter
> > > $ bf_compact $BOGOFILTER_DIR wordlist.db
> > > mv: cannot move `/home/vmail/maildirs/cwilkes/.bogofilter/' to a
> > > subdirectory of itself, `/home/vmail/maildirs/cwilkes/.bogofilter/.old/.bogofilter'
> > 
> > A "mv" failure sounds like permission problems with the parent
> > directory.  What does 
> > 
> >   ls -la /home/vmail/maildirs/cwilkes/.bogofilter/.old
> > 
> > report?
> 
> I had to create that directory for the first time around.  I would try
> and do it again but I'm getting the following errors:
> 
>   $ bf_compact $BOGOFILTER_DIR wordlist.db
>   /home/vmail/maildirs/cwilkes/.bogofilter/wordlist.db.new: No such file or directory
> 
> which keeps on repeating.

I can't reproduce the problem.  I just tried:

   mkdir 0109.d
   bogoutil -l 0109.d < wordlist.txt
   bf_compact 0109.d

and ended up with directories 0109.d and 0109.d.old, which is correct.

Add "-x" at the end of bf_compact's first line and run the command again.
 
> > > Can I use the wordlist.df found in bf_compact.$$/ ?
> > 
> > Not by itself.  A Berkeley DB environment includes _all_ the files in
> > the directory.  As long as _NO_ bogofilter process is running, you can
> > cheat and use "cp -f bf_compact.$$/* $BOGOFILTER_DIR".
> 
> Just did that -- what should I do with all the log.* files?

They can be removed by running "bogoutil --db-prune=$BOGOFILTER_DIR". 
If that doesn't get them, try "db_archive -h $BOGOFILTER_DIR"

> > > I was also able to do a
> > >    bogoutil -d wordlist.db | bogoutil -l wordlist.db.new
> > > without the lock problem.  However I sometimes get these errors:
> > >   $ mv wordlist.db.new wordlist.db
> > >   $ bogofilter -B $files -v
> > >   /home/vmail/maildirs/cwilkes/.bogofilter/wordlist.db.new: No such file or directory
> > > which is a little odd -- a simple grep for "new" in the $BOGOFILTER_DIR
> > > doesn't show anything referencing the new wordlist file.
> 
> I take that back, but I'm not sure how to read the log.* file:
>   $ grep --text \.new log.0000000062 | strings
>   wordlist.db.new
>   wordlist.db.new
>   wordlist.db

grep is just showing that wordlist.db.new is a string in your wordlist,
which is not too surprising.

The log.* files are journal files belonging to BerkeleyDB.  There
shouldn't be any need to read them.

> > That's an odd one.  Bogofilter's DEBUG macros (activated by "-x ...")
> > should help us.  What's the output of command:
> > 
> >    echo this is a test | bogofilter -x d -vv -C 
> 
> $ echo this is a test | bogofilter -x d -vv -C

...[snip]...

I was expecting to see an attempt to open "wordlist.db.new".  Evidently
something's changed so that isn't happening.  Drat!

FWIW, I've attached a patch for the bf_util scripts that verifies the
directory parameter really names a directory.  I don't think it'll help
you, but it won't hurt :-)

Regards,

David


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 192.txt
URL: <http://www.bogofilter.org/pipermail/bogofilter/attachments/20050109/dfa70ed6/attachment.txt>