LMDB v0.9.23 bug (Was: Re: (Was: Re: DB backend support for lmdb?))

[Steffen Nurpmeso]

Hello Matthias.

I was in a hurry on Saturday, please excuse the short message and
the messed up message subject.

Steffen Nurpmeso wrote in <20190309222707.NcLTq%steffen at sdaoden.eu>:
 |Hello once more, and a good Saturday evening, Matthias.
 |
 |As part of a monthly backup i get reproducable LMDB panics with
 |LMDB 0.9.23 when recreating my spam DB:

This is a LMDB bug: it looses the environment size upon
transaction abort unless there was a successful transaction in
between.

Anyway, it can be worked around, we simply need to keep track of
our own idea of the database size, and use that if it is larger
than the one that LMDB reports.  (We are growing the database, so
even if another process would regulary shrink (is that at all
possible?), then we still need to grow dependent upon our own idea
of the database size.)

All tests pass (3 skipped) with the fix as below, and i can load
my dump as normal.  It would be nice to see this fix being
committed, thank you!

Ciao, Matthias.

diff --git a/bogofilter/src/datastore_lmdb.c b/bogofilter/src/datastore_lmdb.c
index 681db24..bfa5080 100644
--- a/bogofilter/src/datastore_lmdb.c
+++ b/bogofilter/src/datastore_lmdb.c
@@ -86,6 +86,9 @@
 # error "Required LMDB version: 0.9.22 or later (0.9.11 may do, but untested)"
 #endif
 
+#ifndef MAX
+# define MAX(A,B) ((A) < (B) ? (B) : (A))
+#endif
 #define UNUSED(x) ((void)(x))
 
 enum a_bflm_flags{
@@ -105,6 +108,7 @@ struct a_bflm{
     MDB_dbi bflm_dbi;
     uint32_t bflm_flags;
     size_t bflm_maxkeysize; /* mdb_env_get_maxkeysize() */
+    size_t bflm_dbsize;     /* LMDB bug: forgets env size after txn abort */
 #ifndef a_BFLM_FIXED_SIZE
     struct a_bflm_txn_cache *bflm_txn_cache;    /* Stack thereof */
 #endif
@@ -483,6 +487,11 @@ jredo_txn:
     mdb_env_set_mapsize(bflmp->bflm_env, 0);
     /* no error defined */mdb_env_info(bflmp->bflm_env, &envinfo);
     i = envinfo.me_mapsize;
+    /* LMDB v0.9.23 bug: forgets the environment size upon TXN abort time
+     * unless a successful TXN has been seen */
+    if(bflmp->bflm_dbsize > i)
+        i = bflmp->bflm_dbsize;
+
     if((size_t)-1 - i >= a_BFLM_GROW * 2){
         i += a_BFLM_GROW / 10;
         i = (i + (a_BFLM_GROW - 1)) & ~(a_BFLM_GROW - 1);
@@ -492,11 +501,13 @@ jredo_txn:
         emsg = "DB size too large";
         goto jerr1;
     }
+
     e = mdb_env_set_mapsize(bflmp->bflm_env, i);
     if(e != MDB_SUCCESS){
         emsg = "mdb_env_set_mapsize()";
         goto jerr1;
     }
+    bflmp->bflm_dbsize = i;
 
     /* Recreate transaction */
     e = mdb_txn_begin(bflmp->bflm_env, NULL, 0, &bflmp->bflm_txn);

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)