[long] Recovery handling in TXN branch

[David Relson]

Matthias,

The phrase that comes to mind is "ick.  what a pain."  I've got some
ideas.  Though I'm not sure how useful they'll be, I'll include them:

The BerkeleyDB background page at
http://www.sleepycat.com/docs/ref/transapp/app.html seems oriented
towards long-running processes, i.e. processes for which a big startup
cost (like running recovery every time) is acceptable.  Since bogofilter
may run multiple times per second that approach doesn't seem a good fit.

Your locking ideas are comprehensive and complex.  Being complex
increases the chances of problems.  Something simpler may well be
appropriate.

Most bogofilter runs are for message scoring which only requires
database reading.  Writing the database when registering messages is the
unusual case.  It seems that bogofilter needs a solution that protects
those writes.

As you know my mail server runs postfix and procmail.  /etc/procmailrc
uses procmail's file locking when it runs "bogofilter -u" and this has
been reliable.  I wonder if we can do something comparable for
transaction locking.  Here's an idea:

Create a bogolock program that gets the global bogofilter lock, then
forks to bogofilter or bogoutil, and waits for completion.  If the
forked process is successful, all is well and the lock is released.  If
the forked process fails, then bogolock can take the appropriate
recovery actions.

Such a control program can be simple, hence bullet proof.

Years ago when I was coding telecom daemons that needed to run 24x7
non-stop, my first effort involved a watchdog process.  Occasionally the
watchdog ended up killing a live daemon and getting all tangles up.  The
final, working solution was having the program fork with the forked
process doing all the work and the parent process waiting for the death
of the forked process so it could be restarted.  Having the simple
control level worked and worked well.

Another approach is running bogofilter as a daemon.  This would be a
client-server design, as has been suggested in the past, and would have
the advantage of the bogofilter daemon running 24x7 which would run
faster since database opening wouldn't be needed for every message.
Cached reading would also be faster.  The server could be bogofilter
itself or it could be a database layer used by bogofilter and bogoutil
or the server.  Again, fork() could be used to control the daemon to
keep it running.

David