Bug#293207: bogofilter: last two versions caused db errors

[Micah Anderson]

On Sun, 13 Feb 2005, Karl Schmidt wrote:

> 
> >
> >The only thing I can think of is that perhaps when you upgrade your
> >exim is still delivering files and things get in a confused state. If
> >you were actually removing and recreating your /etc/bogofilter
> >directory contents, then you would of course need to stop exim in this
> >process to keep this from happening.
> >
> >Micah
> 
> I'm pretty sure I turned exim off by hand before I ran the script - (I will 
> add it to my scrip when I test the next release).
> 
> Even if I failed to turn it off shouldn't the locking prevent such a 
> problem? If not - that would be a bug in my mind. Not a bad idea to test - 
> I could see it getting corrupted but working only to fail after some use.

No, because of permission problems. This is common with DB files,
especially with DB .log files. If the Debian-exim user suddenly cannot
read/write/execute one of the files in the database environment
because it is owned by the root user, then database corruption ensues.
This is not a problem with bogofilter at all, but with permissions on
your system. You can argue until you are blue in the face that it is a
problem with berkeley DB, but it is simply a permissions problem. If a
user who is not root does transactions to a database that this user
has access to, and then (due to the BDB configuration) a new
transaction log has to be created and the existing logfiles rotated,
there will be trouble if one of those .log files is owned by another
user other than the one doing the transaction. This is one of the most
common problems with Subversion using a BDB backend, setting up
permissions is paramount to keeping the DB from being corrupt.

> More I think about this - I don't think exim could do that as the script I 
> run as root creates root owned db files that the Debian-exim user couldn't 
> talk to.
> But, the other way around would be possible - Where exim creates the db and 
> then the script adds to the db and possibly creates some log files with the 
> root ownership that the Debian-exim user can't see.

This is exactly the problem. If your /etc/bogofilter directory has
files that are owned by root, and are not properly permissioned and
exim attempts to do transactions and cannot operate in the database
environment freely, you will get corruption. This is easily solved by
doing a db_recover, or simply making the permissions correct (a common
solution is to set the group sticky bit on the directory, or you could
run your bogofilter seeding as the Debian-exim user).

> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
> 
> Creates some log files with -rw-r--r--  1 root        root
> 
> Note the different owner AND permissions -- ok it is using the users create 
> mask. -- Hmm should that really be the case? Could this scramble the db??

Absolutely... this is *the most common* cause of BDB database
corruption. If exim continues to run, it will soon want to do
operations on the database. It can do so many operations on the
database before a new transaction log is created, it'll be able to
read your root owned -rw-r--r-- files, but if it needs to write,
rename, move or anything to those files, BAM! you'll get DB
corruption. A very simple corruption that can be fixed with
db_recover, the DB wont be "scrambled", but the environment is screwed
and needs to be hand fixed.

> #!/bin/bash
> wajig stop exim4
> # yes my db directory should really be in var some place - but it is in /etc
> rm /etc/bogofilter/* -f
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2003
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
> chown  Debian-exim.Debian-exim /etc/bogofilter/*
> chmod go-r /etc/bogofilter/*
> wajig start exim4
> 
> If this is the cause of the problem, it still leaves a question of how 
> bogofilter should handle running as a different user than the db files?

This is a BDB setup problem, not a bogofilter problem really (and
should not be a grave Debian bogofilter bug as a result). Bogofilter
should run as Debian-exim in this scenario, so it isn't running as a
different user than the db files. As mentioned above, the most common
solution for this for a Subversion BDB environment where multiple
developers are needing to check-in files, who all have different
umasks, is to force the umasks to be something sane (022), but each
user in the same group and set the group stickybit on the database
directory.

Micah