Apparent Memory error in BF and Mutt

[Matthias Andree]

Am 06.07.2016 um 15:20 schrieb Brian McCullough:
> Wrong sender address, again.

Seems to have worked this time...

> It has been about a decade since I last used Bogofilter, so I don't
> remember a lot.

Apparently your file system remembered more than was useful for the
newer software version :-)

> One is an "overloading" of the Save function in Mutt, the other is an
> separate command in Mutt.

These look safe. I'm not quoting them here.

> Wed Jul 06 [bdmc at mail2] ~/$ bogofilter -V
> bogofilter version 1.2.4
>     Database: Berkeley DB 5.3.28: (September  9, 2013) AUTO-XA

> Wed Jul 06 [bdmc at mail2] ~/$ uname -a
> Linux mail2 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u2
> (2016-06-25) x86_64 GNU/Linux
> 
> 
> If it matters, this is a Xen VM.

Just wanted to be sure it's somewhat mainstream. I've been running
bogofilter in VirtualBox VMs, so that's not an issue.

>> And if you're using default directories, the output of
>>
>>   env LC_ALL=C ls -lab $HOME/.bogofilter $HOME/.bogofilter.cf
> 
> 
> Hmmm.  I think that I see a potential problem.
> 
> I told you that it had been a long time.  I am going to remove
> .bogofilter and see if that makes any difference.  B-)
> 
> 
> I will show you the top and bottom of the listing, but not all of it.
> 
> Wed Jul 06 [bdmc at mail2] ~/$ env LC_ALL=C ls -lab $HOME/.bogofilter
> $HOME/.bogofilter.cf
> ls: cannot access /home/bdmc/.bogofilter.cf: No such file or directory
> /home/bdmc/.bogofilter:
> total 441320
> drwx--S---   2 bdmc bdmc    20480 May 15  2005 .
> drwxr-xr-x 112 bdmc bdmc    12288 Jul  6 09:02 ..
> -rw-------   1 bdmc bdmc    16384 Apr  6  2005 __db.001
> -rw-------   1 bdmc bdmc  5251072 Apr  6  2005 __db.002
> -rw-------   1 bdmc bdmc    98304 Apr  6  2005 __db.003
> -rw-------   1 bdmc bdmc  4063232 Apr  6  2005 __db.004
> -rw-------   1 bdmc bdmc    16384 Apr  6  2005 __db.005

These __db.00* are the files that clogged up the Berkeley DB library,
and is what Berkeley DB is picky about.  It's been some ten years, and I
guess it's been Berkeley DB 4.2 or older that created these files back
then, and the log.* files (more on the latter below), and these __db.00*
make up the database "environment".  Berkeley DB 5.3 can't use the older
__db.00* files.

I suspect - provided that the database (the *.db file) itself wasn't
corrupted - that just removing these five files would have sufficed.

> -rw-rw-r--   1 bdmc bdmc        0 Apr  3  2005 lockfile-d
> -rw-rw-r--   1 bdmc bdmc     1024 May 16  2005 lockfile-p
> -rw-rw-r--   1 bdmc bdmc  1048576 Apr  3  2005 log.0000000001
> -rw-------   1 bdmc bdmc  1048576 Apr  3  2005 log.0000000002
[...]
> -rw-------   1 bdmc bdmc  1048576 May 16  2005 log.0000000404
> -rw-------   1 bdmc bdmc 18800640 Jul  6 09:02 wordlist.db
> Wed Jul 06 [bdmc at mail2] ~/$ 

These log files contain the transaction log, after an upgrade, Berkeley
DB will just create a new one if the format changed.

> 
> 
> 
> 
> 
>  
>> It would be good to see debugging output, from these commands:
>>
>>   bogofilter -QQ
> 
> Wed Jul 06 [bdmc at mail2] ~/$ bogofilter -QQ
[...]

These looked sound, except for the known complaint.


> Here's the LS after removing the ancient history:
> 
> Wed Jul 06 [bdmc at mail2] ~/$ env LC_ALL=C ls -lab $HOME/.bogofilter
> $HOME/.bogofilter.cf
> ls: cannot access /home/bdmc/.bogofilter.cf: No such file or directory
> /home/bdmc/.bogofilter:
> total 116
> drwx------   2 bdmc bdmc   4096 Jul  6 09:10 .
> drwxr-xr-x 113 bdmc bdmc  12288 Jul  6 09:09 ..
> -rw-------   1 bdmc bdmc 102400 Jul  6 09:11 wordlist.db
> Wed Jul 06 [bdmc at mail2] ~/$ 
> 

I do not see __db.00* and log.* files.  This means that the database is
not in transactional mode, meaning that it can get corrupted in an
application or computer crash, and would then have to be restored from
backup, or recreated from scratch.

If you desire to upgrade this to a transactional database such that it
can be recovered from software and some classes of hardware crashes,
it suffices to run the NEXT bogoutil or bogofilter command that access
the database with --db-transaction=yes, such as:

	bogofilter --db-transaction=yes </dev/null

This is a one-time operation. After that you'll see __db.001, .002, .003
files, lockfile-d and lockfile-p files, and on the next WRITE access
(registration, or running with -u if there wasn't an "unsure" result),
you'll also see a log.0000000001 file.

(The recovery can be run through bogoutil --db-recover=DIRECTORY, where
DIRECTORY normally is $HOME/.bogofilter)