DB corruption within minutes

[Gyepi SAM]

On Sat, Jan 11, 2003 at 06:07:32AM +0100, Matthias Andree wrote:
> Gyepi SAM <gyepi at praxis-sw.com> writes:
> 
> >> Could bogofilter use an external lock file of some sort?  Would this help 
> >> or would it create even more trouble and complication?
> >
> > Sure, if the problem is with locking the db files.
> 
> OK, I have 16 hours of serialized bogofilter now, with flock() in my
> .mailfilter surrounding it, that means I have at most one bogofilter
> running at a time. I has gone without corruption, with near 900 mails
> passing through it, which formerly would have killed the db in less than
> two hours.

Sounds pretty compelling.

> I have a strong suspect: our db locking. It always hits my goodlist.db,
> never my spamlist.db (but then again, there's still more ham than spam
> here), and bogofilter running in -u)pdate mode.

Surely, it must hit the spamlist.db sometime. Also, we lock the goodlist.db
first becuase that's first in the list.
 
> I have added t.lock2 and made minor fixes to bogofilter.

I noticed. I also change the grind loop to run for 1000 iterations,
and found no problems.
 
> I'm wondering if we can do things the way we do. We release the lock
> before closing the data base, which does not look right. BDB has
> internal caches (like stdio) that need to be flushed by ->close before
> we can let go of the lock, so the ordering would likely have to be
> db_close, db_lock_release, but db_close kills the handle, so all of this
> should be more tightly integrated.

The only solutions I can think of are
1. call open (2) on the database ourselves, so we have a handle to lock
2. use an external lockfile.
 
> Plus, I believe we cannot release the lock, have someone else update the
> db and then grab the lock again to proceed. The pages may have changed,
> so we have inconsistent cache/disk data.

If we call db_sync() after updating the database but before releasing the lock,
that should fix any syncronization problems of that sort.

-Gyepi