cannot dump wordlist

[David Relson]

On Sat, 3 Dec 2005 22:07:12 -0500
David Relson wrote:

> Matej,
> 
> Two additional ideas.
> 
> 1 - copy wordlist.db to a new directory and try dumping that copy.
> Starting with a fresh directory might help.
> 
> 2 - send your wordlist to me so I can try dumping it.  If you like the
> idea, gzip it and upload it to ftp.bogofilter.org (directory
> pub/incoming) and I'll dump it and get it back to you.
> 
> HTH,
> 
> David

Hi Matej,

I built a copy of bogofilter-0.93.5 with BerkeleyDB-3.2.9 (which is
what you've got) and did some testing with my current wordlist.db which
is 52MB (somewhat larger than your 37MB wordlist).

Without neither bogofilter.cf nor DB_CONFIG, I got the messages you
reported:

   Lock table is out of available locks
   bogoutil: (c_get): Cannot allocate memory
   error dumping tokens!

To run bf_resize, I copied db_archive and db_stat from  
/usr/local/BerkeleyDB.3.2/bin to my bogofilter-0.93.5/src
directory.  This added 2 lines to DB_CONFIG:

  set_lk_max_locks 26096
  set_lk_max_objects 26096

With these values, I was able to successfully run

   bogoutil -C -d wordlist.d/wordlist.db > wordlist.txt

The "-C" option prevents bogoutil from reading a bogofilter.cf file
which might contain conflicting lock or object counts.

Out of curiosity, to trace the derivation of the DB_CONFIG values, I ran
db_stat:

   db_stat -h wordlist.d -d /home/relson/bogofilter/0935/src/wordlist.d/wordlist.db

   53162   Btree magic number.
   8       Btree version number.
   Flags:
   2       Minimum keys per-page.
   4096    Underlying database page size.
   3       Number of levels in the tree.
   1387081 Number of unique keys in the tree.
   1387081 Number of data items in the tree.
   81      Number of tree internal pages.
   8998    Number of bytes free in tree internal pages (97% ff).
   13048   Number of tree leaf pages.
   750548  Number of bytes free in tree leaf pages (99% ff).
   0       Number of tree duplicate pages.
   0       Number of bytes free in tree duplicate pages (0% ff).
   0       Number of tree overflow pages.
   0       Number of bytes free in tree overflow pages (0% ff).
   0       Number of pages on the free list.

Value 26096 is double 13048 which is the file size (52M) divided by the
database page size (4096).

HTH,

David