bogofilter randomly fails to find db's

[Roger 'Rocky' Vetterberg]

David Relson wrote:

> On Fri, 09 Jan 2004 20:38:21 +0100
> "Roger 'Rocky' Vetterberg" <listsub at 401.cx> wrote:
> 
> 
>>Hi again,
>>
>>David Relson wrote:
>>
>>
>>>On Fri, 09 Jan 2004 18:47:58 +0100
>>>"Roger 'Rocky' Vetterberg" <listsub at 401.cx> wrote:
>>>
>>>Welcome Rocky,
>>>
>>>
>>>>Hi list.
>>>>
>>>>I recently installed bogofilter on a FreeBSD 4.9/Sendmail 8.12.9p2 
>>>>server. Im using a series of procmail rules and shellscripts to feed
>>>
>>>>spam into bogofilter to train it.
>>>>Every 30 minutes, a shellscript checks a certain imap folder for new
>>>
>>>>messages, and if it finds any, it feeds it to bogofilter with the 
>>>>following command: /usr/local/bin/bogofilter -s < $basedir/$junk
>>>>The folder contains between 0 and 20-25 messages each time,
>>>
>>>depending >on the amount of spam recieved in the last 30 minutes.
>>>
>>>>Usually, it works like a charm, but on what seems like random 
>>>>occations it fails with the following error:
>>>>
>>>>Can't open files 'spamlist.db' and 'goodlist.db' in directory 
>>>>'/var/lib/bogofilter',
>>>>error #2 - No such file or directory.
>>>>
>>>>Remember to register some spam and ham messages before you
>>>>use bogofilter to evaluate mail for its probable spam status!
>>>>Learned from 1 message(s) (1 message(s) examined).
>>>
>>>
>>>What version of bogofilter are you using?  What files do you have in
>>>/var/lib/bogofilter?  Do you have one database file, i.e.
>>>wordlist.db, or two (spamlist.db and goodlist.db)?
>>>
>>
>>Here is some output that might help.
>>All commands were typed in exactly as shown as soon as the previous 
>>command was finished.
>>
>>rambo# bogofilter -V
>>bogofilter version 0.15.7
>>     Algorithms: Fisher Graham Robinson
>>     Database: BerkeleyDB (4.0.14), combined
>>Copyright (C) 2002 Eric S. Raymond
>>
>>bogofilter comes with ABSOLUTELY NO WARRANTY. This is free software, 
>>and you
>>are welcome to redistribute it under the General Public License. See
>>the COPYING file with the source distribution for details.
>>
>>rambo# whoami
>>rocky
>>rambo# ls -l /var/lib/bogofilter
>>total 29768
>>-rwxrwxrwx  1 rocky  wheel  15171655 Jan  9 00:44 goodlist.db
>>-rwxrwxrwx  1 rocky  wheel  15286272 Jan  9 18:48 spamlist.db
>>rambo# bogofilter -M -d /var/lib/bogofilter -s < .mail/Junkmail
>>Can't open files 'spamlist.db' and 'goodlist.db' in directory 
>>'/var/lib/bogofilter',
>>error #2 - No such file or directory.
>>
>>Remember to register some spam and ham messages before you
>>use bogofilter to evaluate mail for its probable spam status!
>>rambo#
> 
> 
> Hi Rocky,
> 
> Having the problem happen interactively is good - very good.  It'll make
> it easy to know when we get you fixed.
> 
> For the moment, I'd like you to try two things.
> 
> First, since 0.15.7 has been replaced by 0.15.13.1 as the stable release
> can you download the tarball from sourceforge and build it?  That would
> give you the best available code.  I know there have been a variety of
> changes between 0.15.7 and 0.15.13.1, though I don't recall if any might
> apply to your situation.

Since Im running FreeBSD, I rarely downloads tarballs, I use the ports 
system. :)
But you were correct. Running portupgrade showed that there was a new 
  10.15.13,1 version available. However, upgrading did not solve the 
problem. It behaved exactly like the old version.

> Second, having separate databases is the old way of doing things and has
> been deprecated.  It would be best if you'd run "bogoupgrade -d
> /var/lib/bogofilter" which will read the separate databases and will
> create a new, combined database named wordlist.db.  Bogofilter will use
> wordlist.db instead of spamlist.db and goodlist.db.

I tried, but no luck.

rambo# bogoupgrade -d /var/lib/bogofilter
Can't open file '/var/lib/bogofilter/goodlist.db'
error running "bogoutil -d /var/lib/bogofilter/goodlist.db":  at 
/usr/local/bin/bogoupgrade line 158.

> After each of "first" and "second" see if your "bogofilter -s" command
> works.

When bogoupgrade also failed on the db's, I started to believe that 
maybe the db's themselves had something to do with the error.
I renamed both db's, and started feeding spam to bogofilter to build 
up a new db. And guess what, it worked. A new db was created, and with 
this new db I have not seen the error a single time, inspite of heavy 
manual provocation.
My guess is that the db's somehow got corrupted and were unreadable. I 
must say bogofilter had a funny way of letting me know this, but I 
have learned the hard way not to trust what the error messages say.

> We'll get you going !!!

Well, you did...sort of.
My filters seems to run just fine now, the only drawback is that I 
have to start training the filters from the beginning again, but I can 
live with that.

Thanks for all your assistance, you've been most helpfull!

--
R