BerkeleyDB lock manager [was .count files]

[Adrian Otto]

Gyepi,

> A db file can actually hold multiple named databases, so I was
> planning to use a single db file for all the lists and their counts.

Please don't do that yet. It adds some complexity, and will cause the file
locking to become more impacted. If we leave it the way it is now, but
simply eliminate the ".count" files, it's possible for an update of each
wordlist to happen simultaneously by two separate processes. If we combine
them into a single file, then processes will queue on the file locks more
often.

We could start using the BerkeleyDB lock manager, and this locking problem
will be mitigated. If we do use the BerkeleyDB lock manager, we could stop
using file-level locks. We'll need to learn more about how the lock manager
works. These questions come to mind:

1) If the BerkeleyDB lock manager is used, can processes on multiple hosts
operate on the same database, and be mutually aware of what data is locked
at what time? With our current code, fcntl file locking gives us this
functionality.

2) What version of the BerkeleyDB system did the lock manager show up in? If
someone has version 2 of the library, the current code should work. If we
add the lock manager support, will we be restricting our support to systems
that have version 3 or 4?

When answering these questions, we should remember the importance of the
ability for external applications to interoperate with the wordlist data
files directly. We should also remember that we want bogofilter to be as
portable as possible. If using it means upgrading to a newer version of the
BerkeleyDB code to get locking support, we might be compromising this goal.

Thanks,

Adrian