bogofilter randomly fails to find db's

[Roger 'Rocky' Vetterberg]

Hi again,

David Relson wrote:

> On Fri, 09 Jan 2004 18:47:58 +0100
> "Roger 'Rocky' Vetterberg" <listsub at 401.cx> wrote:
> 
> Welcome Rocky,
> 
>>Hi list.
>>
>>I recently installed bogofilter on a FreeBSD 4.9/Sendmail 8.12.9p2 
>>server. Im using a series of procmail rules and shellscripts to feed 
>>spam into bogofilter to train it.
>>Every 30 minutes, a shellscript checks a certain imap folder for new 
>>messages, and if it finds any, it feeds it to bogofilter with the 
>>following command: /usr/local/bin/bogofilter -s < $basedir/$junk
>>The folder contains between 0 and 20-25 messages each time, depending 
>>on the amount of spam recieved in the last 30 minutes.
>>Usually, it works like a charm, but on what seems like random 
>>occations it fails with the following error:
>>
>>Can't open files 'spamlist.db' and 'goodlist.db' in directory 
>>'/var/lib/bogofilter',
>>error #2 - No such file or directory.
>>
>>Remember to register some spam and ham messages before you
>>use bogofilter to evaluate mail for its probable spam status!
>>Learned from 1 message(s) (1 message(s) examined).
> 
> 
> What version of bogofilter are you using?  What files do you have in
> /var/lib/bogofilter?  Do you have one database file, i.e. wordlist.db,
> or two (spamlist.db and goodlist.db)?
> 

Here is some output that might help.
All commands were typed in exactly as shown as soon as the previous 
command was finished.

rambo# bogofilter -V
bogofilter version 0.15.7
     Algorithms: Fisher Graham Robinson
     Database: BerkeleyDB (4.0.14), combined
Copyright (C) 2002 Eric S. Raymond

bogofilter comes with ABSOLUTELY NO WARRANTY. This is free software, 
and you
are welcome to redistribute it under the General Public License. See the
COPYING file with the source distribution for details.

rambo# whoami
rocky
rambo# ls -l /var/lib/bogofilter
total 29768
-rwxrwxrwx  1 rocky  wheel  15171655 Jan  9 00:44 goodlist.db
-rwxrwxrwx  1 rocky  wheel  15286272 Jan  9 18:48 spamlist.db
rambo# bogofilter -M -d /var/lib/bogofilter -s < .mail/Junkmail
Can't open files 'spamlist.db' and 'goodlist.db' in directory 
'/var/lib/bogofilter',
error #2 - No such file or directory.

Remember to register some spam and ham messages before you
use bogofilter to evaluate mail for its probable spam status!
rambo#

> Your "bogofilter -s ..." command is running in registration mode and
> will create a database if it isn't already present.  Stated differently,
> "bogofilter -s" shouldn't fail because of a missing file, so the message
> is incorrect :-(
> 
> I'm assuming that you also have bogofilter running to score incoming
> messages and that the training task is running into a database locking
> problem.  Bogofilter has a retry loop to keep trying (for a while) when
> it can't open the database for writing (as needed by "-s") because the
> database is open for reading (as needed for scoring incoming messages).

You are correct, bogofilter is invoked via procmail everytime an email 
is recieved. However, the amount of email trafic on this machine is 
moderate at best, and sofar only a testing account is using 
bogofilter, so I really think that the odds of a incoming email is 
locking the database exactly when this happens is extremely low.
During the session pasted above I made sure the account did not 
recieve any emails, and the error still occured.

> I suggest that you add the "-l" (log) option to your bogofilter commands
> for scoring and training.  Then you'll be able to check the system log
> to see if multiple copies of bogofilter are running into a locking
> problem.

I've added the -l flag to all places where I invoke bogofilter. I will 
let it run for a while and then examine the logs to see if something 
shows up. Good tip, thanks.


--
R