bogofilter randomly fails to find db's

[David Relson]

On Fri, 09 Jan 2004 18:47:58 +0100
"Roger 'Rocky' Vetterberg" <listsub at 401.cx> wrote:

Welcome Rocky,

> Hi list.
> 
> I recently installed bogofilter on a FreeBSD 4.9/Sendmail 8.12.9p2 
> server. Im using a series of procmail rules and shellscripts to feed 
> spam into bogofilter to train it.
> Every 30 minutes, a shellscript checks a certain imap folder for new 
> messages, and if it finds any, it feeds it to bogofilter with the 
> following command: /usr/local/bin/bogofilter -s < $basedir/$junk
> The folder contains between 0 and 20-25 messages each time, depending 
> on the amount of spam recieved in the last 30 minutes.
> Usually, it works like a charm, but on what seems like random 
> occations it fails with the following error:
> 
> Can't open files 'spamlist.db' and 'goodlist.db' in directory 
> '/var/lib/bogofilter',
> error #2 - No such file or directory.
>
> Remember to register some spam and ham messages before you
> use bogofilter to evaluate mail for its probable spam status!
> Learned from 1 message(s) (1 message(s) examined).

What version of bogofilter are you using?  What files do you have in
/var/lib/bogofilter?  Do you have one database file, i.e. wordlist.db,
or two (spamlist.db and goodlist.db)?

Your "bogofilter -s ..." command is running in registration mode and
will create a database if it isn't already present.  Stated differently,
"bogofilter -s" shouldn't fail because of a missing file, so the message
is incorrect :-(

I'm assuming that you also have bogofilter running to score incoming
messages and that the training task is running into a database locking
problem.  Bogofilter has a retry loop to keep trying (for a while) when
it can't open the database for writing (as needed by "-s") because the
database is open for reading (as needed for scoring incoming messages).

I suggest that you add the "-l" (log) option to your bogofilter commands
for scoring and training.  Then you'll be able to check the system log
to see if multiple copies of bogofilter are running into a locking
problem.

With the requested info (logging and bogofilter), we should be able to
take this further.

David

> The command mentioned above is executed every 30 minutes 24 hours a 
> day (provided there is spam in the folder), and it fails on average 
> maybe 3 times each day. Sometimes it may fail two times in a row, but 
> more often the failures are spread evenly across the day.
> 
> Of course, I have checked that the directory does indeed exist, all 
> permissions are set correctly and the db is writeable. Running the 
> command manually never produces these errors, it has so far only 
> happened when executed from crontab.
> I have even modified the script so that it checks that the directory 
> and both db's really exists before running the bogofilter command, and
> 
> sofar this check has never failed, but still bogofilter complains not 
> even a second later that they dont exist.
> 
> The only idea Ive had is that perhaps the previous process may still 
> be running and therefor somehow puts a lock on the db, but when 
> timeing the process I have never seen it running for more then 2 
> minutes, so the process started 30 minutes ago shold have finished 
> long before the next run is invoked.
> 
> Im simply stumped by this and totally out of ideas. Does anyone have 
> any idea what might cause this?
> 
> Please cc any replies to me since Im not subscribed to the list.
> 
> -- 
> R