[long] Recovery handling in TXN branch

[Matthias Andree]

On Mon, 16 Aug 2004, Gyepi SAM wrote:

> I hate to say it, especially since I introduced it to the project, but Berkeley DB
> is a difficult and problematic system to work with.

Difficult? No. Complex? Yes.

> From personal experience and other projects I have used, it appears
> that trying to use any of its complex features leads to headaches. A
> common example: Look at often rpm hangs on newer redhat systems and
> must be killed off.

Never seen that happen on SuSE 9.1 (which uses RPM 4.1.1).

I have acquired some experience with BerkeleyDB and I do believe I
understand every situation where it can lock up.

> Frequently, it turns out that an rpm instance
> locked the database and failed to unlock it for whatever reason so all
> subsequent attempts to use the database block until the database lock
> files are manually deleted.

That happens with the "Concurrent" and "Transactional" style on unclean
shutdown.

Anyways, that is the exact problem I'm trying to solve: detect unclean
shutdowns so we can kill off processes and run recovery. I haven't yet
implemented this in spite the BDB documentation requesting that we do
so. And it's about time this gets made up for. I'm confident we'll see
no lockups when that crash detector code is in place and working.

> What we need here is a small, fast
> embedded database with transactional capabilities. Ideally, we should
> support only a single database instead of the 3 or 4 that currently
> exists.

I'm fine with dropping TDB and QDBM. QDBM has been a quality nightmare,
and TDB - at least the way your code uses its locking - is painfully
slow.

> Perhaps it's time to investigate an alternative database system. I recommend
> SQLite (http://www.sqlite.com/). It has all the desired features with, AFAIK,
> none of the drawbacks of BDB.

Yet it must parse SQL. I'd also thought about how well PostgreSQL might
work, but I'm not sure how we stuff halfways binary data (tokens) into
that data base. I'd think SQL is a bit oversized.

-- 
Matthias Andree

NOTE YOU WILL NOT RECEIVE MY MAIL IF YOU'RE USING SPF!
Encrypted mail welcome: my GnuPG key ID is 0x052E7D95 (PGP/MIME preferred)