bogofilter randomly fails to find db's

[David Relson]

On Fri, 09 Jan 2004 20:38:21 +0100
"Roger 'Rocky' Vetterberg" <listsub at 401.cx> wrote:

> Hi again,
> 
> David Relson wrote:
> 
> > On Fri, 09 Jan 2004 18:47:58 +0100
> > "Roger 'Rocky' Vetterberg" <listsub at 401.cx> wrote:
> > 
> > Welcome Rocky,
> > 
> >>Hi list.
> >>
> >>I recently installed bogofilter on a FreeBSD 4.9/Sendmail 8.12.9p2 
> >>server. Im using a series of procmail rules and shellscripts to feed
> >
> >>spam into bogofilter to train it.
> >>Every 30 minutes, a shellscript checks a certain imap folder for new
> >
> >>messages, and if it finds any, it feeds it to bogofilter with the 
> >>following command: /usr/local/bin/bogofilter -s < $basedir/$junk
> >>The folder contains between 0 and 20-25 messages each time,
> >depending >on the amount of spam recieved in the last 30 minutes.
> >>Usually, it works like a charm, but on what seems like random 
> >>occations it fails with the following error:
> >>
> >>Can't open files 'spamlist.db' and 'goodlist.db' in directory 
> >>'/var/lib/bogofilter',
> >>error #2 - No such file or directory.
> >>
> >>Remember to register some spam and ham messages before you
> >>use bogofilter to evaluate mail for its probable spam status!
> >>Learned from 1 message(s) (1 message(s) examined).
> > 
> > 
> > What version of bogofilter are you using?  What files do you have in
> > /var/lib/bogofilter?  Do you have one database file, i.e.
> > wordlist.db, or two (spamlist.db and goodlist.db)?
> > 
> 
> Here is some output that might help.
> All commands were typed in exactly as shown as soon as the previous 
> command was finished.
> 
> rambo# bogofilter -V
> bogofilter version 0.15.7
>      Algorithms: Fisher Graham Robinson
>      Database: BerkeleyDB (4.0.14), combined
> Copyright (C) 2002 Eric S. Raymond
> 
> bogofilter comes with ABSOLUTELY NO WARRANTY. This is free software, 
> and you
> are welcome to redistribute it under the General Public License. See
> the COPYING file with the source distribution for details.
> 
> rambo# whoami
> rocky
> rambo# ls -l /var/lib/bogofilter
> total 29768
> -rwxrwxrwx  1 rocky  wheel  15171655 Jan  9 00:44 goodlist.db
> -rwxrwxrwx  1 rocky  wheel  15286272 Jan  9 18:48 spamlist.db
> rambo# bogofilter -M -d /var/lib/bogofilter -s < .mail/Junkmail
> Can't open files 'spamlist.db' and 'goodlist.db' in directory 
> '/var/lib/bogofilter',
> error #2 - No such file or directory.
> 
> Remember to register some spam and ham messages before you
> use bogofilter to evaluate mail for its probable spam status!
> rambo#

Hi Rocky,

Having the problem happen interactively is good - very good.  It'll make
it easy to know when we get you fixed.

For the moment, I'd like you to try two things.

First, since 0.15.7 has been replaced by 0.15.13.1 as the stable release
can you download the tarball from sourceforge and build it?  That would
give you the best available code.  I know there have been a variety of
changes between 0.15.7 and 0.15.13.1, though I don't recall if any might
apply to your situation.

Second, having separate databases is the old way of doing things and has
been deprecated.  It would be best if you'd run "bogoupgrade -d
/var/lib/bogofilter" which will read the separate databases and will
create a new, combined database named wordlist.db.  Bogofilter will use
wordlist.db instead of spamlist.db and goodlist.db.

After each of "first" and "second" see if your "bogofilter -s" command
works.

We'll get you going !!!

> 
> > Your "bogofilter -s ..." command is running in registration mode and
> > will create a database if it isn't already present.  Stated
> > differently,"bogofilter -s" shouldn't fail because of a missing
> > file, so the message is incorrect :-(
> > 
> > I'm assuming that you also have bogofilter running to score incoming
> > messages and that the training task is running into a database
> > locking problem.  Bogofilter has a retry loop to keep trying (for a
> > while) when it can't open the database for writing (as needed by
> > "-s") because the database is open for reading (as needed for
> > scoring incoming messages).
> 
> You are correct, bogofilter is invoked via procmail everytime an email
> 
> is recieved. However, the amount of email trafic on this machine is 
> moderate at best, and sofar only a testing account is using 
> bogofilter, so I really think that the odds of a incoming email is 
> locking the database exactly when this happens is extremely low.
> During the session pasted above I made sure the account did not 
> recieve any emails, and the error still occured.

FWIW, My mail server is at the low end of the scale (a 166mhz Pentium
machine running Linux).  It gets approx 700 messages per day, uses
autoupdate ('-u') to update the wordlist for each incoming message and
hourly runs a cronjob to train on any messages that were misclassified
(a rare event) or were classified as "unsure" (which happens several
times a day).  With this combination of slow processor, low mail volume,
and database updates, I've not seen your problem.