Bug#293207: bogofilter: last two versions caused db errors
Matthias Andree
matthias.andree at gmx.de
Wed Feb 2 10:56:26 CET 2005
Karl Schmidt <karl at xtronics.com> writes:
> Clint Adams wrote:
>
>
>> Are you using libdb4.3 4.3.27-1 with the problematic bogofilter
>> versions?
>
> I have:
>
> libdb4.3 4.3.27-1
Please run "bogofilter -V" to check the bogofilter and Berkeley DB
versions, the first two lines are sufficient. Do this with either
bogofilter version. Remember that if you're inadvertently going forth
and back between Berkeley DB versions, your database environment may
break like this. Going backwards isn't supported (so bogoutil -d before
the upgrade, remove the database, downgrade, bogoutil -l), going
forwards requires you to remove the environment _BEFORE_ the update.
I have rewritten parts of README.db after the 0.93.5 release, hence I'm
adding the rewritten version below, perhaps it can help.
--
Matthias Andree
============================================================================
-------------- next part --------------
BERKELEY DB ENVIRONMENT CODE
============================
$Id: README.db,v 1.22 2005/01/27 11:32:28 m-a Exp $
This document does not apply when you are installing a bogofilter
version that has been configured to use the TDB or QDBM data base
managers.
0. Definitions ---------------------------------------------------------
Whenever ~/.bogofilter appears in the text below, this is the directory
where bogofilter keeps its data base by default. If you are overriding
this directory by configuration or environment variables, replace your
actual bogofilter data base directory.
1. Overview ------------------------------------------------------------
This bogofilter version has been upgraded to use the Berkeley DB
Transactional Data Store, to be able to recover a data base after an
application or system crash.
2. Prerequisites and Caveats -------------------------------------------
2.1 Compatibility, Berkeley DB versions
These versions are supported (with all SleepyCat-posted patches applied
- if using a pre-packaged Berkeley DB version, the packager should have
applied the patches, check your vendor's update site regularly):
Sleepycat Software: Berkeley DB 3.1.17: (July 31, 2000)
Sleepycat Software: Berkeley DB 3.2.9: (January 24, 2001)
Sleepycat Software: Berkeley DB 3.3.11: (July 12, 2001)
Sleepycat Software: Berkeley DB 4.0.14: (November 18, 2001)
Sleepycat Software: Berkeley DB 4.1.25: (December 19, 2002)
Sleepycat Software: Berkeley DB 4.2.52: (December 3, 2003)
Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004)
Other 3.x or 4.x versions of Berkeley DB may or may not work but usually
they will.
Berkeley DB versions 4.1 and newer are recommended over the previous
versions, because the newer can detect data corruptions more reliably
(through the use of checksums that detect partially written data base
pages); Berkeley DB 4.2 and 4.3 appear a bit faster under load than 4.1
and older versions.
2.2 Upgrading from non-transactional releases of bogofilter (before 0.93)
NOTE: for updates of Berkeley DB itself, see section 2.6!
Bogofilter should transparently upgrade the existing data base to the
new transactional data base.
For enhanced reliability (only available with Berkeley DB 4.1 or newer),
it is recommended that you dump and reload the database once so that
Berkeley DB adds page checksums. Skip this procedure for Berkeley DB
versions 4.0 or older. For 4.1 and newer, use these commands:
cd ~/.bogofilter
bogoutil -d wordlist.db > wordlist.txt
mv wordlist.db wordlist.db.old
bogoutil -l wordlist.db < wordlist.txt
And if all commands succeeded: rm wordlist.txt wordlist.db.old
NOTE: transactional databases require large lock tables, the exact
size depends on the size of the database. If the lock table isn't
large enough, you may get errors and need to increase the lock table
size, see section 3.2 and perhaps 4.2 when you see this problem.
"bogoutil -d wordlist.db > /dev/null" can be used to check if the lock
table size is sufficient, as it will access the whole data base.
2.3 Recoverability
The ability to recover the data base after a crash (power failure!)
depends on data being written to the disk (or a battery-backed write
cache) _immediately_ rather than delayed to be written later.
Common disk drives in current PCs and MACs are of the ATA or SATA kind
and usually ship with their write cache enabled. They write fast, but
can lose or corrupt up to a few MB of data when the power fails.
Note: This problem is not specific to bogofilter.
It is possible to sacrifice a bit bit of the the write speed and get
reliability in turn, by switching off the disk's write cache (see
appendix A for instructions).
Switching the write cache off may however adversely affect the
performance below acceptable levels, particularly for large writes such
as recording live audio or video data to hard disk.
If performance is degraded too much, consider getting a separate disk
drive and using one for fast writes (with the write cache on) and one
for reliable writes (with the write cache off, for bogofilter, mail
servers and other applications that need survive a power loss without
data loss).
2.4 Choosing a file system
If your computer saves the data on its own disk drive (a "local file
system"), Berkeley DB should work fine. Such file systems are ext2, ext3,
ffs, jfs, hfs, hfs+, reiserfs, ufs, xfs.
Berkeley DB Transactional and Concurrent data stores do not work
reliably with a networked file system for various technical reasons.
AFS, CIFS, Coda, NFS, SMBFS fall into this category.
Strictly speaking, with Berkeley DB 4.0 and older versions, the data base
block size must be written atomically. The bogofilter maintainers are
not currently aware of a file system that meets this requirement and is
production quality at the same time.
2.5 Making a snapshot backup
The transactional data store is no good if the disk drive has become
inaccessible (which happens after some months or years with every
drive), so you _must_ back up your data base regularly (see the
db_archive utility for additional documentation of a "hot" backup),
bogofilter cannot, of course, guess data that got lost through a hard
drive fault.
When copying or archiving directory contents, be sure to copy or archive
the *.db files FIRST, BEFORE archiving/copying the log.* files, this
is needed to let the database copy or archive remain recoverable.
You can use the bf_tar script for convenience. It requires the pax
utility (UNIX standard for over a decade now) and writes a tar archive
to stdout, and it can optionally remove inactive log files before or
after writing the tarball.
Run bf_tar without arguments to see its synopsis.
2.6 Updating Berkeley DB version underneath bogofilter
When upgrading the Berkeley DB library to a new version, or recompiling
bogofilter to use a newer version, two things in the on-disk data format
can change, generally speaking: the database format (we use the BTree
access method), the log file format, or both.
You need a "log file upgrade" if at least one of these conditions is
true:
- you upgraded Berkeley DB from a 3.X version to a 4.Y version
- you upgraded Berkeley DB from 4.0 or 4.1 to 4.2 or 4.3
- you upgraded Berkeley DB from 4.2 to 4.3.
If you need a log file upgrade, the upgrade procedure is:
(NOTE: DO NOT UPGRADE BERKELEY DB OR BOGOFILTER UNTIL STEP 4!)
1. shut down your mail system,
2. run 'bogoutil --db-remove-environment ~/.bogofilter' (for each user)
(this implies running forced recovery first)
3. archive the database for catastrophic recovery (take a backup)
4. install the new Berkeley DB version, recompile bogofilter (unless
using a binary package), install the new bogofilter
5. restart your mail system.
If you've been using Berkeley DB 3.0 (only supported with bogofilter
versions 0.17.2 and older) and are about to update to any newer 3.X or
4.X version, you need a database format upgrade. You can either:
- dump the wordlists to a text file, then update Berkeley DB
and bogofilter, then load the wordlists again;
or
- use the db_upgrade utility to upgrade the databases in place
(this is dangerous and must not be interrupted, backup first!)
3. Use and troubleshooting ---------------------------------------------
3.1 LOG FILE HANDLING
The Berkeley DB Transactional Data Store uses log files to store data
base changes so that they can be rolled back or restored after an
application crash.
The logs of the transactional data store, log.NNNNNNNNNN files of up to
1 MB in size (in the default configuration), can consume considerable
amounts of disk space and many users wish to purge or compress these log
files. For the simple removal of log files that are no longer active,
you can run 'bogoutil --db-prune ~/.bogofilter'. It will checkpoint the
database (to make some more log files inactive) and then remove the
inactive log files.
If you wish to keep and for instance compress (gzip) the log files, you
can safely use Berkeley DB's db_checkpoint and db_archive utilities,
which should be run in this order:
- First, use db_checkpoint. It migrates written-ahead data from the log files
into the data base, and places a checkpoint which will speed up data
base recovery, for instance after a premature bogofilter abort.
- db_archive allows to identify log files that are no longer in use so
that you can compress or remove them.
Note that if you plan to remove log files, you must make snapshot
backups that contain both the *.db and log.* files (in this order!).
Referral: Berkeley DB's db_archive documentation contains suggestions
for several backup strategies.
3.2 LOCK TABLE EXHAUSTION
One common failure case is known:
Problem: Operations that affect large parts of the data base or the data
base as a whole (bogoutil usually) may require many locks and
exhaust the maximum number of locks or the maximum number of
locked objects that the Berkeley DB environments support.
Symptom: Operations abort with "out of memory" although the machine has
plenty of RAM and/or swap.
Operations report lock or object table exhaustion and abort.
Cause: Natural data base growth.
Fix: Resize the lock tables. Just run "bf_resize ~/.bogofilter" and
that's it. This script will modify the ~/.bogofilter/DB_CONFIG
file and run recovery to perform the reize. It'll resize the
lock and object tables to twice of what is currently in use.
You may need to adjust these figures before larger updates or to conserve
disk space. You will need up to one lock per data base page and a bit
of headroom for future training -- see section 4.2 below to determine
the size of the data base and data base page. After changing these
set_lk_max* lines in DB_CONFIG, run 'bogoutil --db-recover ~/.bogofilter'
to make the change effective.
3.3 RECOVERY AND FAILED RECOVERY
The recovery procedures should be tried in the order shown in this
section. If you aren't willing to experiment much, but have kept
sufficient spam and ham that you can easily and quickly retrain
bogofilter from scratch, read only sections 3.3.1 and 3.3.5, skipping
subsections 3.3.2 to 3.3.4.
3.3.1 Regular recovery
Bogofilter and related bogo* utilities will automatically detect when
regular recovery is needed and run it. This process is transparent,
the user will usually be aware this happens.
This process needs the *.db file and the corresponding _active_ log
files.
It is possible to trigger regular recovery by running
bogoutil --db-recover ~/.bogofilter
although this should not be needed.
If this fails, remove the __db.*, *.db and log.* files,
restore from the latest snapshot backup (see section 2.5) and force
recovery as shown in the previous paragraph.
3.3.2 Catastrophic recovery
If regular recovery fails after severe damage to hardware, filesystem,
database files, you can attempt to run catastrophic recovery. If log
files have been damaged, catastrophic recovery may not work.
This may need *all* log files from the backup and is therefore not
available if log files have been pruned.
To run catastrophic recovery, replace the log files from your archives,
then run:
bogoutil --db-recover-harder ~/.bogofilter
If this fails, read on.
3.3.3 Last-resort recovery method #1: nuke the environment
This recovery methods do not guarantee you are getting all of your
database, you may already have lost part or all of your data when this
is required, and you may lose recent updates to the database and corrupt
it. Only attempt this methods if the regular and catastrophic recovery
methods have failed or were unavailable.
To use this method:
1. remove the __db.* and log.* files
2. run: bogoutil -v --db-verify ~/.bogofilter/wordlist.db
3a. if that printed OK, watch carefully if bogofilter performs to
the standards you are used to.
3b. if verify failed, read the next section.
3.3.4 Last-resort recovery method #2: salvage the raw .db file
This recovery methods do not guarantee you are getting all of your
database, you may already have lost part or all of your data when this
is required, and you may lose recent updates to the database and corrupt
it. Only attempt this methods if the regular and catastrophic recovery
methods have failed or were unavailable.
To try this method:
db_dump -r ~/.bogofilter/wordlist.db > ~/.bogofilter/wordlist.txn
rm ~/.bogofilter/{__db.*,log.*,wordlist.db}
db_load ~/.bogofilter/wordlist.db < ~/.bogofilter/wordlist.txn
3.3.5 No recovery possible?
We're sorry. This should happen really rarely. There's nothing left to
try, so you need to retrain from scratch.
First, remove the database and environment, type:
rm ~/.bogofilter/{__db.*,log.*,wordlist.db}
Then retrain from scratch with the usual bogofilter -n and bogofilter -s
commands.
4. Other Information of Interest ---------------------------------------
4.1 GENERAL INFORMATION
Berkeley DB keeps some additional statistics about locking, caching and
their efficiency. These can be obtained by running the db_stat utility:
db_stat -h ~/.bogofilter -d wordlist.db # data base statistics
db_stat -h ~/.bogofilter -e # environment statistics
db_stat -h ~/.bogofilter -c # lock statistics - needed for lock resizing
db_stat -h ~/.bogofilter -m # buffer pool statistics
db_stat -h ~/.bogofilter -l # log statistics
db_stat -h ~/.bogofilter -t # transaction statistics
Note that statistics may disappear when the data base is recovered. They
will reappear after running bogofilter and are the more reliable the
more often bogofilter has been used since the last recovery.
You MUST NOT remove files named __db.NNN and log.NNNNNNNNNN manually
from the ~/.bogofilter directory.
REMOVING THESE FILES CAUSES DATABASE CORRUPTION!
(there is one exception, see below)
These can contain update data for the data base that must be still
written back to the wordlist.db file - this happens when there are many
concurrent processes alongside a registration process.
Exception: after reading the Berkeley DB documentation for the
db_archive utility and using that utility, you may be able to remove
the INACTIVE log.NNNNNNNNNN files, or use 'bogoutil --db-prune ...'.
This may be necessary to reclaim disk space, but you must strictly
adhere to the Berkeley DB documentation lest you risk your data base
become unrecoverable in case of trouble.
WARNING: If you need to copy data base files,
DO NOT USE cp, BUT DO USE dd instead and give it a block size
that matches the data base's block size, which can be found by
running db_stat with -d option as shown above.
A bf_copy script is provided for your convenience, it can
optionally omit the inactive log files from the copy.
4.2 SPECIFIC INFORMATION ON RESIZING THE LOCK TABLES
In all the commands shown below, replace the ~/.bogofilter path by the
name of the directory holding your wordlist.db file.
a. Determine the data base size:
ls -l ~/.bogofilter/wordlist.db
b. Determine the data base page size:
db_stat -h ~/.bogofilter/ -d wordlist.db
The relevant line has "database page size"
c. The number of locks and lock objects needed is the data base size
divided by the data base page size, rounded up generously, for example:
(output from step a)
-rw-r--r-- 1 joe users 15360000 2004-05-11 12:25 wordlist.db
(output from step b)
53162 Btree magic number.
8 Btree version number.
Flags:
2 Minimum keys per-page.
4096 Underlying database page size.
3 Number of levels in the tree.
...
Hence: 15360000 / 4096 = 3750
d. Round up, 4096 may be adequate if you train seldomly, use
higher values if you train often or in preparation of training on a
large mailbox. Higher values make your lock region, usually __db.004,
larger, but allow for larger data bases. Lower values save disk space
but may require you to to resize the lock region more often.
e. Use this rounded-up figure for both of the the two DB_CONFIG file
lines mentioned in section 3.1
f. run 'bogoutil --db-recover ~/.bogofilter'
Bogofilter will re-create the lock tables automatically
the next time it is run.
4.3 INTERESTING CONFIGURATION PARAMETERS FOR DB_CONFIG
A "DB_CONFIG" file that is in your bogofilter home directory, usually
~/.bogofilter, can be used to configure the data base behavior. Some
options take effect immediately, some need 'bogoutil --db-recover' to
become effective.
Here is a list of interesting settings, put one at a line, omitting the
leading spaces and hyphen:
SIZING OPTIONS:
- set_cachesize G B C
(valid in Berkeley DB 3.1 - 4.3, requires recovery to change)
sets the cache size to G gigabytes plus B bytes which are spread out
in C equally sized caches (all figures are natural numbers). You
cannot configure caches smaller than 20 kB, Berkeley DB will increase
all caches under 500 MB by 25%, and you must provide at least one
cache per 4 GB.
This option takes precedence over Bogofilter's db_cachesize option!
Example: set_cachesize 0 60000000 6
will create six caches sized 12500000 bytes (12.5 MB, +25% applied)
- set_lk_max_locks 12345
- set_lk_max_objects 12346
(valid in Berkeley DB 3.2 - 4.3, requires recovery to change)
see sections 3.2 and 4.2 of this document for how to set the numbers.
Choosing these figures too high wastes disk space,
choosing them too low may cause bogofilter applications to abort.
- set_lg_max 250000
(valid in Berkeley DB 3.1 - 4.3, takes effect immediately)
this option configures the maximum log file size, in bytes, before
Berkeley DB starts a new log file. The default is 1 MB.
SAFE OPTIONS:
- set_flags DB_DIRECT_DB
(valid in Berkeley DB 4.1 - 4.3, takes effect immediately)
this option turns off system buffering of *database* files, to avoid
double caching of data. NOT SUPPORTED ON ALL PLATFORMS!
- set_flags DB_DIRECT_LOG
(valid in Berkeley DB 4.1 - 4.3, takes effect immediately)
this option turns off system buffering of *log* files, to avoid
double caching of data. NOT SUPPORTED ON ALL PLATFORMS!
- set_flags DB_DSYNC_LOG
(valid in Berkeley DB 4.3, takes effect immediately)
this option can increase performance on some systems (and decrease on
other systems), by using the O_DSYNC POSIX flag rather than a
separate function to flush the logs.
- set_flags DB_NOMMAP
(valid in Berkeley DB 3.2 - 4.3, takes effect immediately)
this option can reduce memory consumption at run time, particularly
with large databases, at some cost of performance
- set_flags DB_REGION_INIT
(valid in Berkeley DB 3.2 - 4.3, takes effect immediately)
this option causes all shared memory regions to be "page faulted"
into core memory at application start and written at data base
creation. This can improve performance under load, and it allows
bogofilter or bogoutil to allocate disk space in advance, to avoid
some out-of-space conditions later on.
- set_verbose DB_VERB_CHKPOINT
(valid in Berkeley DB 3.1 - 4.2, takes effect immediately)
- set_verbose DB_VERB_DEADLOCK
(valid in Berkeley DB 3.1 - 4.3, takes effect immediately)
- set_verbose DB_VERB_RECOVERY
(valid in Berkeley DB 3.1 - 4.3, takes effect immediately)
- set_verbose DB_VERB_WAITSFOR
(valid in Berkeley DB 3.1 - 4.3, takes effect immediately)
these verbose flags cause extended output for long-lasting
operations, ...CHKPOINT prints location information when searching
for checkpoints, ...DEADLOCK prints information on deadlock
detection, ...RECOVERY prints information during recovery and
...WAITSFOR prints the waits-for table during deadlock detection.
UNSAFE OPTIONS - these may impair robustness/recoverability of the data base
- set_flags DB_LOG_AUTOREMOVE
(valid in Berkeley DB 4.2 - 4.3, takes effect immediately)
this option lets Berkeley DB automatically remove log files as soon
as they are no longer needed. It will prevent "catastrophic"
recovery (because the log files are missing), but regular recovery
should still work.
- set_flags DB_TXN_NOSYNC
(valid in Berkeley DB 3.3 - 4.3, takes effect immediately)
if set, the log is not written or synchronously flushed at commit
time. In case of an application or system crash, the last few
registrations can be lost.
- set_flags DB_TXN_WRITE_NOSYNC
(valid in Berkeley DB 4.1 - 4.3, takes effect immediately)
if set, the log is written, but not synchronously flushed at commit
time. In case of a system crash, the last few registrations can be
lost.
DANGEROUS OPTIONS - these can improve performance, but should be avoided
- set_flags DB_TXN_NOT_DURABLE
(valid in Berkeley DB 4.2, takes effect at next application start,
replaced by DB_LOG_INMEMORY for version 4.3)
this option prevents writing into log files. In case of application
or system crashes, the data base can become corrupt, and large
registrations can exhaust the log buffer space and then fail.
- set_flags DB_LOG_INMEMORY
(valid in Berkeley DB 4.3, takes effect at next application start)
this option prevents the writing of any log files to disk. In case of
application or system crashes, the data base can become corrupt, and
large registrations can exhaust the log buffer space and then fail.
After a crash, verify the data base (see below) and if it is corrupt,
restore from backup or rebuild from scratch.
4.4 VERIFYING DATABASES TO CHECK FOR CORRUPTION
To verify that the database is intact, type:
bogoutil --db-verify ~/.bogofilter/wordlist.db
There should be no output. If there are errors, you must recover your
database, see section 3.3 for methods.
A. Switching the disk drive's write cache off and on -------------------
A.1 Introduction
You need to determine the name of the disk device and its type.
Type "mount", you'll usually get an output that contains lines like
these; find the "on /home" if you have it, if you don't check for "on
/usr" if you have it, or finally, resort to looking at the "on /" line.
>From this line, look at the left hand column, usually starting with /dev.
If you have FreeBSD, skip to section A.3 now.
A.2 Switching the write cache off or on in Linux
In this line you've found (see previous section A.1), you'll usually find
something that starts with /dev/hda, /dev/hde or /dev/sda in the left
hand column of that line, you can ignore the trailing number. /dev/hd*
means ATA, /dev/sd* means SCSI.
If the drive name starts with /dev/hd, type the following line, but
replace hda by hde or what else you may have found:
/sbin/hdparm -W0 /dev/hda
(replace -W0 by -W1 to reenable the write cache)
If your drive name starts with /dev/sd, use the graphical scsi-config
utility and add a blank the device name on the command line; for
example:
scsi-config /dev/sda
You can "try changes" (they will be forgotten the next time the computer
is switched off) or "save changes" (settings will be saved permanently);
you can use the same utility to restore the previous setting or load
manufacturer defaults. Skip to section 2.4.
What is this scsi-config?
The scsi-config command is a Tk script, delivered with the scsiinfo
package. At the time of writing, scsiinfo can be found at
ftp://tsx-11.mit.edu/pub/linux/ALPHA/scsi/scsiinfo-1.7.tar.gz .
For users who don't run X on their mail servers, there is also a
command-line utility, scsiinfo, in the package. Setting parameters
with scsiinfo is a bit hairy, but the following sequence worked for two
of us who tried it (back up your drive first):
# get current disk settings and turn off the write cache
# (substitute the appropriate device for /dev/sda in all these commands)
parms=`scsiinfo -cX /dev/sda | sed 's/^./0/'`
# write the parameters back to the hard drive's current settings
# this needs to be put in a boot script
scsiinfo -cXR /dev/sda $parms
# if you don't want to put this in a boot script, you can alternatively
# save the parameters to the hard drive's settings area:
scsiinfo -cXRS /dev/sda $parms
You did back up your drive before trying that, right? :)
A.3 Switching the write cache off in FreeBSD
Have you read section A.1 already? You should have.
In this line you've found (see section A.1), you'll usually have a line
that starts with /dev/ad0, /dev/wd0 (either means you have ATA) or
/dev/da0 (which means you have SCSI).
If you have ATA, add the line
hw.ata.wc="0"
to /boot/loader.conf.local, shut down all applications and reboot. (To
revert the change, remove the line, shut down all applications and
reboot.)
If you have SCSI, you'll need to decide if you want the setting until the next
reboot, or permanent (the permanent setting can be changed back, don't worry).
In either case, omit the leading /dev and trailing s<NUMBER><LETTER> parts
(/dev/da0s1a -> da0; /dev/da4s3f -> da4). Replace da0 by your device name in
these examples, and leave out the part in parentheses:
camcontrol modepage da0 -m8 -e -P0 (effective until computer is switched off)
camcontrol modepage da0 -m8 -e -P3 (save parameters permanently)
camcontrol will open a temporary file with a WCE: line on top. Edit the
figure to read 0 (cache disabled) or 1 (cache enabled), then save the
file and exit the editor.
More information about the bogofilter
mailing list