New bogofilter TXN snapshot available.

[David Relson]

On Sun, 31 Oct 2004 02:58:34 +0200
Torsten Veller wrote:

> * David Relson <relson at osagesoftware.com>:
> > Torsten Veller wrote:
> > > I don't think so, David. I think the wordlist must be rebuild (?).
> > 
> > I'm not yet convinced. 
> 
> > In the command you've used, bogofilter will look for wordlist.db in
> > the normal place, i.e. ~/.bogofilter/wordlist.db, rather than the
> > current directory.  Instead of "bogofilter -v", try "bogofilter -v
> > -x d -v". 
> 
> Directories were the same. PWD was ~/.bogofilter. Sorry, should have
> said this.

That would have helped.  I rarely remember to include every possible
detail, so I can't fault you.  Anyhow, I'm glad you know what to do!

> > DB_ENV->open(home=test-0.92.99.d/)
> > db_version: Header version 4.2, library version 4.2
> > [pid 10683] DB->open(db=0x8072e70, file=wordlist.db, database=NIL,
> >   type=1, flags=0x1000000=DB_AUTO_COMMIT , mode=0664) -> 0
> >   Successful return: 0
> 
> 
> Excerpt from the dump to wordlist.txt:

Looks pretty reasonable.  It's a bit different from mine because mine is
all english, so the accented characters aren't present.

> /-----------
> Áàíêà 2 0 20040921
> Áñ°__¸´Â 1 0 20040909
> Á÷Á¢ 1 0 20040909
> Áø½ÉÀ¸·Î 2 0 20040909
>  Now 0 1 20040915
>  output 0 1 20040915
>  sfftobmp 0 1 20040915
>  tiff2ps 0 1 20040915
>          0 1 20040915
>         apt-get 0 1 20040915
>         cd 0 1 20040915
>         debuild 0 1 20040915
>         http 0 1 20040915
>         sudo 0 1 20040915
> ÂÈÐÓÑÎÂ 1 0 20040909
> ÂÎÑÑÒÀÍÎÂËÅÍÈÅ 1 0 20040909
> Âàì 2 0 20040921
> Âàñ 2 0 20040921
> \-----------
> 
> > I wrote a test script to do reproduce what you did.  It assumes that
> > you have 4 executables available:
> > 
> >   bogofilter-0.92.8  and bogoutil-0.92.8
> >   bogofilter-0.92.99 and bogoutil-0.92.99
> > 
> > It creates a directory "test-0.92.99.d" and uses it for all
> > operations. I've attached it as file test-0.92.99.sh and its output
> > as test-0.92.99.out
> > 
> > After testing "echo abc | bogofilter -x d -vv" to verify your
> > wordlist path, run "echo abc | bogofilter -d . -v" to show what
> > happens when using the wordlist you build.  Then run my script. 
> > Hopefully this will make clear exactly what is happening.
> 
> I doubled line 18:
> echo abc | bogofilter-0.92.99 -C -d $BOGODIR -v
> echo
> echo abc | bogofilter-0.92.99 -C -d $BOGODIR -v
> 
> Output of the script is attached.
> 
> > It _may_ be that you're having a Berkeley DB problem since we're
> > using different versions.  You've got 4.1.25 and I'm running 4.2.52.
> >  However,
> > I don't really think that's what's happening.
> 
> db upgraded - no problems with the script. I can test my upgrade
> process tomorrow again.
> 
> bogofilter version 0.92.99.cvs
>     Database: BerkeleyDB (4.2.52)
> 
> > I hope you're your still online so we can determine what's really
> > happening! 
> 
> Now it's really bedtime.

Your bedtime is far past mine.  I've not been up at 3:00 AM for years.

> -- 
> + BOGODIR=test-0.92.99.d
> + WORDLIST=test-0.92.99.d/wordlist.db
> + '[' -d test-0.92.99.d ']'
> + rm -rf test-0.92.99.d
> + bogofilter-0.92.8 -V -C
> bogofilter version 0.92.8
>     Database: BerkeleyDB (4.1.25)
> Copyright (C) 2002-2004 Eric S. Raymond,
> David Relson, Matthias Andree, Greg Louis
> 
> bogofilter comes with ABSOLUTELY NO WARRANTY.  This is free software,
> and you are welcome to redistribute it under the General Public
> License.  See the COPYING file with the source distribution for
> details.
> 
> + echo
> 
> + mkdir test-0.92.99.d
> + bogoutil-0.92.8 -l test-0.92.99.d/wordlist.db
> + ls -l test-0.92.99.d
> total 2636
> -rw-r--r--  1 tv users 2695168 31. Okt 02:17 wordlist.db
> + echo
> 
> + bogofilter-0.92.99 -V -C
> bogofilter version 0.92.99.cvs
>     Database: BerkeleyDB (4.1.25)
> Copyright (C) 2002-2004 Eric S. Raymond,
> David Relson, Matthias Andree, Greg Louis
> 
> bogofilter comes with ABSOLUTELY NO WARRANTY.  This is free software,
> and you are welcome to redistribute it under the General Public
> License.  See the COPYING file with the source distribution for
> details.
> 
> + echo
> 
> + echo abc
> + bogofilter-0.92.99 -C -d test-0.92.99.d -v
> checksum error: catastrophic recovery required
> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> wordlist.db: pgin failed for page 0
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> fatal region error detected; run recovery
> PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
> fatal region error detected; run recovery
> Can't open file 'wordlist.db' in directory 'test-0.92.99.d'.
> error #22 - Invalid argument.
> 
> Make sure that the BerkeleyDB version this program is linked against
> can handle the format of the data base file (after updates in
> particular) and that your NFS locking, if applicable, works.

This _may_ be the key!  We're using different versions of Berkeley DB -
4.1.25 and 4.2.52.  Is there any possibility that you can rebuild and
test with the newer version?

I think we're at the point where more database expertise is needed.  As
you know, Matthias fills that role for bogofilter.  That's a very good
thing as he knows much more than I do.  

Anyhow, since he's away for the weekend, we need to wait for his return.
 Given the severity of what you've encountered, I'm going to hold the
release of 0.93.0 until we know what's going on.

David

P.S.  I'm glad you got the cvs source, tested, and encountered this
problem _before_ the release.  Thank you.