catastrophic recovery required

[David Relson]

On Sat, 5 Feb 2005 17:02:57 +1100
Ben Finney wrote:

> Howdy all,
> 
> Today a run of bogofilter tells me "catastrophic recovery required".
> 
> =====
> $ bogoutil -v --db-recover-harder /home/bignose/.bogofilter
> checksum error: page 2004: catastrophic recovery required
> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> wordlist.db: pgin failed for page 2004
> unable to create/retrieve page 2004
> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> Recovery function for LSN 47 489705 failed on forward pass
> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> PANIC: fatal region error detected; run recovery
> bogoutil: DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery
> To recover, run: bogoutil -v --db-recover "/home/bignose/.bogofilter"
> 
> $ bogofilter --version
> bogofilter version 0.93.5
>     Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004)
> [...]
> =====
> 
> So, the recovery command itself fails, and recommends I run recovery.
> 
> What are my options from here?  How can I further diagnose this?

G'day Ben,

What a way to start the day -- an error on the far side of the world, in
the land down under.  Uh, boy!

Did anything unusual happen?  Power failure, operating system crash,
bogofilter crash, ...? 

What's your environment?  Operating system, processor, networking, ...?
Did you build bogofilter yourself?  What options?

Matthias is the Berkeley DB expert, so I'll let him supply the final
answer.  In the meantime, copy your environment so there's opportunity
for a post mortem.  "bf_copy" is the tool for the job.

Next, you can try "bogoutil -d" to dump the wordlist and see how much
you get.  The results will be alphabetized, so you can judge
completeness by how close you get to "zzzzzzz".  If there's a high level
of completeness, you can build a new wordlist from the dump output.
Note: losing 20% (maybe even 50%) of the alphabet will have _some_
impact on accuracy, but it won't be disastrous since the part of the
database you have will have quantities of known good and known spam
words in the same proportion as your original database.  Stated another
way, a partial database will be usable.  Of course you'll want to do
training because you will have lost part of it.

Alternate to "bogoutil -d", you can try the actual Berkeley DB tools --
db_verify, db_recover, db_dump -- and see what happens.

In the worst case, you'll need to start all over, though I expect the
bogoutil technique will get you going.

HTH,

David