(Was: Re: DB backend support for lmdb?)

[Steffen Nurpmeso]

Hello once more, and a good Saturday evening, Matthias.

As part of a monthly backup i get reproducable LMDB panics with
LMDB 0.9.23 when recreating my spam DB:

  $ </tmp/spam.db \
  /tmp/bogofilter.tar_bomb_git/bogofilter/src/bogoutil -l x.lmdb
  ONE
  SET_MAPSZIE 1048576 -> 16777216
  NEW MAPSIZE: 16777216
  ANUHIGN FINE!
  ONE
  SET_MAPSZIE 1048576 -> 16777216
  NEW MAPSIZE: 16777216
  ANUHIGN FINE!
  ONE
  SET_MAPSZIE 1048576 -> 16777216
  NEW MAPSIZE: 16777216
  ANUHIGN FINE!
  ONE
  SET_MAPSZIE 1048576 -> 16777216
  NEW MAPSIZE: 16777216
  ANUHIGN FINE!
  RETRIES 4 FAILS
  bogoutil[17156]: LMDB[17156]: db_set_dbvalue(), mdb_cursor_put(): -30792, MDB_MAP_FULL: Environment mapsize limit reached

So the debug instrumentations above show that the resize actually
failed, in that it succeeded but does not get honoured (with
0.9.23 that is), de facto.  So this finally convinces me that LMDB
does not truly like resizing the map.  You possibly remember that
struggled almost a week with crashes until i finally gave up
calling mdb_env_set_mapsize() upon DB creation time, which made
them disappear.  Later such calls did not cause any harm (until
today).  The crashes were also the reason for the second
implementation which uses fixed size.

I mean, i am using this without just any problems ever since, and
do a monthly dump/restore to "vacuumize" the DB.  That with LMDB
0.9.23 (the first dump/restore with it, 0.9.22 before) we enter
the condition that anything works as expected,
mdb_env_set_mapsize() succeeds but LMDB f...s it up is pretty
unfortunate.

What i have done locally now is to simply do the below, aka choose
the fixed size implementation which does not use the (my wonderful
and pretty cool, well, average and la la) replay cache, and with
that even 0.9.23 can be convinced the recreate my spam DB:

-rw-r-----  1 steffen steffen      8192 Mar  9 22:23 wordlist.lmdb-lock
-rw-r-----  1 steffen steffen 162926592 Mar  9 22:23 wordlist.lmdb
-rw-r-----  1 steffen steffen       576 Mar  9 22:31 x.lmdb-lock
-rw-r-----  1 steffen steffen  83546112 Mar  9 22:31 x.lmdb
The dump:
-rw-r----- 1 steffen steffen 62993221 Mar  9 17:04 /tmp/spam.db

So maybe bogofilter should switch to the fixed size
implementation, too?

diff --git a/bogofilter/src/datastore_lmdb.c b/bogofilter/src/datastore_lmdb.c
index 681db24..42c1bd5 100644
--- a/bogofilter/src/datastore_lmdb.c
+++ b/bogofilter/src/datastore_lmdb.c
@@ -41,7 +41,7 @@
  */
 
 /* Alternative implementation: fixed DB size */
-/*#define a_BFLM_FIXED_SIZE (ULONG_MAX >> (ULONG_MAX != UINT_MAX ? 22 : 1))*/
+#define a_BFLM_FIXED_SIZE (ULONG_MAX >> (ULONG_MAX != UINT_MAX ? 22 : 1))
 
 /* mdb_env_set_maxreaders() */
 #define a_BFLM_MAXREADERS 7

Ciao, Matthias!
And a nice weekend i wish.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)