Ideas wanted for TXN and Concurrent store recovery handling

[Matthias Andree]

Tom Anderson <tanderso at oac-design.com> writes:

> On Wed, 2004-07-28 at 06:15, Matthias Andree wrote:
>> Well, switching to a daemon might enable us to ease use in the network
>> (as long as the load level remains acceptable), but will still be
>> subject to the same problem, if the daemon goes down, it will be facing
>> the same problems as the standalone bogofilter process today - how and
>> when to recover, guard against concurrent recovery or recovery
>> underneath use, and all that.
>
> The daemon won't start if there is another instance already running, and
> it won't start any children if it is doing recovery, which solves the
> concurrent use/recovery problem.

The daemon wouldn't need to spawn children IMO, it can be
single-threaded to reduce locking contention in the data base, and
might, after all, eliminate the need for any kills to resolve a deadlock.

> You could have a lock flag in the wordlist which is set when the
> daemon starts and is unset when it cleanly exits.  Thus, if the daemon
> starts and this flag is set, then it should do recovery.

A "dirty flag" inside the DB is the problem, the access of the DIRTY
flag may already lock the application up without actually reading it. It
would need to be in a separate file.

>> That's not my primary concern now, but how to avoid a bogofilter process
>> (that might also be the daemon process) from waiting for an event that
>> can never occur.
>
> Such as what?  Query children via IPC to check their status; if there's
> no response, kill it.
>
> I never claimed to have all the answers, just an idea, as requested.  

Yup, thank you :-)

My current idea is to have an empty file inside the .bogofilter
directory that can serve as fcntl anchor (might be DB_CONFIG or the
wordlist.db file, too, or the directory itself but that may not work
because some operating systems don't permit write access opens on
directories which is necessary for fcntl).

Any running bogofilter process would claim a shared lock and then try to
upgrade to an exclusive lock once. Only one process can hold the
exclusive lock, and that process would then be in charge of running
recovery. The remaining question is how does the process know if regular
or catastrophic recovery is needed?  Neither does harm, but the latter
can take a whole age or two...

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)