DB backend support for lmdb?

Tue May 29 20:51:11 CEST 2018

Ahoi.

Matthias Andree wrote:
 |Am 28.05.2018 um 23:57 schrieb Steffen Nurpmeso:
 |>|Note that -u incurs writes so is prone to whatever consistency and
 |>|durability models the database uses - and it will hurt with LMDB too due
 |>|to its "one writer only", so you serialize those processes if you use
 |>|"-u" just as much as you would with -n or -s (unless you have lots of
 |>|unsures, when -u is useless).
 |> 
 |> Hmm, likely the test was not thorough enough.  Sorry.  Yes, of
 |> course, but likely that because the same messages had been passed
 |> nothing new happened (but counter increments?); the sqlite version
 |> did change the size, and the DB one not, that was definetely true.
 |
 |It depends on whether the write causes page overflows and new pages to
 |be created, or not.  Certainly using SQLite is overkill because we still
 |used it as a key/value store... but it was a user request at the time
 |and I have yet to see reports about other trouble than it not being the
 |fastest or writing the smallest databases.

Please, i have absolutely nothing against sqlite, i used it
myself.  It may even be interesting to have the possibility to
look into the tables and access them with SQL statements, for
those who have interest in that.

Maybe because i have it in git, and all those repos are reduced to
a "null" branch to save backup space.  If i need something i check
out the "master" and compile that.  And git does not restore file
times when the checkout happens.  Ok, so maybe also my fault.

 |If it only hurts developers, that's a non-issue, and I paid attention to
 |write the configure cache out in strategic places.  You know that when I
 |started developing bogofilter c. 15 years ago, computers and disk drives
 |were a lot slower.
 |
 |./configure -C # ...  advised. :-)

Yes i know the former.  The latter not, i will try it; especially
today many projects with submodules require multiple configuration
runs, i hope that helps there.  I have added that to all
configure runs in my (extern.)code.arena makefile.  Thanks!

 |The autotools stuff is in place, works reliably, has a rich feature set,
 |and with recent automake implementations, "make check" tests run in
 |parallel.  Do that with SSD on a modern octocore computer and see it fly
 |in spite of a recursive Makefile structure, or perhaps with /tmp a
 |RAMDISK and then make check BF_TESTDIR=/tmp -- I wouldn't want to use an
 |early Raspberry Pi as development platform though.  Deployment is
 |another matter.

hmmhmm, yes, well.. ;)

 |>|It depends. SQLite uses the same extension, .db, and Berkeley DB has two
 |>|modes, one is the plain old (which is not ACID compliant) and has only
 |>|the one [wordlist].db file - I advise against using that unless you can
 |>|recreate the database anytime from saved spam/ham corpora.
 |> 
 |> Ah!  This i did not know, i have always worked with packages until
 |> just recently, after finding the space and performance issue
 |> i compiled on my own the first time.  Sorry, sorry; i have read
 |> the bogofilter manual once when i had thrown away the homebrew
 |> junk mail code from the MUA i maintain, in order to create
 |> a working environment for the new mail code -- in summer 2013.
 |
 |bogofilter -V tells you the database it is using, and there's doc/README.db.
 |
 |>|The other is the transactional mode (advertised as this Berkeley DB
 |>|Transactional Data Store) that writes additional files, for instance,
 |>|trivially:
 |>|
 |>|__db.001
 |>|__db.002
 |>|__db.003
 |>|lockfile-d
 |>|lockfile-p
 |>|log.0000000001
 |>|wordlist.db
 |> 
 |> Yes, that is what i knew, and this is what i get after
 |> recompilation with --enable-transactions.  Thanks for the
 |> information!
 |
 |And perhaps after running any registering stuff with
 |--db-transaction=yes once. And perhaps undoing the same registration.

It is a powerful tool, and i am using only 5 percent of it.
(That command line option is not to be seen in a manual page.)

 |> It seems to me there is quite a lot of context that i did not know
 |> about.  So i think implementing LMDB support will not be a quick
 |> shot if done right, i need to read a lot of documentation from
 |> LMDB and source code from Bogofilter.  So it may take a little bit
 |
 |LMDB should be less of a hassle - and on second thought, it might be
 |easier to start the implementation off after reading the Berkeley DB
 |*and* Kyotocabinet (or Tokyocabinet) implementations to see the simpler
 |ones.

Will do, soon.

 |> longer until i can provide a patch -- nonetheless, i am definetely
 |> very interested in LMDB support for bogofilter, if doable, because
 |> it is very small (the raw AlpineLinux code package is 90KB,
 |> whereas DB is 1.6MB; the cloned repo is 1.2MB, whereas the 5.3.28
 |> DB tar ball unpacked in git is 31MB), and the code is also open
 |> and openly maintained.  And Postfix supports LMDB as a replacement
 |> for DB out of the box, too.  All this is very desirable to me.
 |
 |Repo size of a support library isn't normally a relevant metric, but
 |this is a valid point, as is its license:
 |
 |   text          data     bss     dec     hex filename
 |  80510          1504       8   82022   14066 /usr/lib64/liblmdb.so

Runtime is much smaller here, too:

  #?0[steffen at essex nail.git]$ size /usr/lib/liblmdb.so 
     text    data     bss     dec     hex filename
    69680    1344      80   71104   115c0 /usr/lib/liblmdb.so
  #?0[steffen at essex nail.git]$ size /usr/lib/libdb.so
     text    data     bss     dec     hex filename
  1549515   38744      64 1588323  183c63 /usr/lib/libdb.so

I am looking forward for this.
Ciao, and thanks for the informations!

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)