Ideas wanted for TXN and Concurrent store recovery handling

[Matthias Andree]

Pavel Kankovsky <peak at argo.troja.mff.cuni.cz> writes:

> On Wed, 28 Jul 2004, Matthias Andree wrote:
>
>> The daemon wouldn't need to spawn children IMO, it can be
>> single-threaded to reduce locking contention in the data base, and
>> might, after all, eliminate the need for any kills to resolve a deadlock.
>
> Deadlocks can be avoided when you do all updates withing a single
> transaction in the predefined order (e.g. lexicographical order of
> keys).

This currently happens for efficiency reasons, we use a btree and that
profits from data locality but registering a larger box consumes
considerable amounts of memory then.

>> My current idea is to have an empty file inside the .bogofilter
>> directory that can serve as fcntl anchor (might be DB_CONFIG or the
>> wordlist.db file, too, or the directory itself but that may not work
>> because some operating systems don't permit write access opens on
>> directories which is necessary for fcntl).
>
> This should work...as long as the underlying fs implements fcntl() locks
> in a correct way (NFS is infamous for having locking of, ehm, questionable
> quality but I don't think BDB itself would work well in such an
> environment).

The .db file across NFS is fine, but the logging/locking/memory
environment isn't too happy about networked file systems in general. It
has certain atomicity requirements that NFS doesn't provide.

>> The remaining question is how does the process know if regular
>> or catastrophic recovery is needed?  Neither does harm, but the latter
>> can take a whole age or two...
>
> As far as I know, the catastrophic recovery is intended for situations
> when db files are lost or corrupted and the database is being rebuilt
> from archived snapshots and logs.

...which may happen after a crash, the same situation that causes locks
that haven't been cleared. During normal operation, we don't get into a
situation that requires recovery at all.

I liked the dirty flag idea (to avoid unnecessary recovery) but don't
see how that would work.

> Ergo I think the right approach is to run the normal recover and give up
> if it fails (alternatively, it might be possible to fail-over to a read
> only backup db).

In that situation, it will fail with an error. I wonder if we need a
"corrupt" marker so that not every new bogofilter process started after
the previous figured it cannot recover will retry but can exit with
error early.

> --Pavel Kankovsky aka Peak  [ Boycott Microsoft--http://www.vcnet.com/bms ]
> "Resistance is futile. Open your source code and prepare for
> assimilation."

s/assimilation/evolution/ :)

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)