LMDB 1.0 / 0.9 database incompatibility

[Matthias Andree]

Am 06.07.26 um 19:58 schrieb Steffen Nurpmeso:
> Hello Matthias.
>
> Matthias Andree via bogofilter-dev wrote in
> <d19429bc-83a0-4c32-8fa5-c6a88c0a9cec at gmx.de>:
> |Am 04.07.26 um 21:43 schrieb Steffen Nurpmeso:
> |> just saw 3cee1c59bc937 -- interesting! I have not tried LMDB 1.0
> |> yet, but saw some thread talking on it being entirely incompatible
> |> DB-wise. Ie, 0.9 databases are simply not supported at all!
> |> Whereas i understand Howard Chu a bit when he says
> |>
> |> The on-disk file format has changed in LMDB 1.0 and versions 0.9
> |> and 1.0 are mutually incompatible. You must use v0.9 #mdb_dump
> |> to export your old DBs, and import with v1.0 #mdb_load if you
> |> want to migrate your existing data to use LMDB 1.0. There is
> |> no support in LMDB 1.0 for operating directly on v0.9 DB files.
> |> Including such support would only bloat the library so it will
> |> not be done.
> |>
> |> it causes some trouble "downstream", and that is only the packager
> |> side.. A transition period would have been nice.
> |>
> |> I think i myself follow certain advices and create completely new
> |> packages, ie keep bogofilter-lmdb and add bogofilter-lmdb1.
> |
> |For FreeBSD that would make sense for a transitional period; worst case
> |we need to make a copy of the ultimate LMDB 0.9.x into bogofilter's
> |sources and use it to build a feature-stripped "bogoutil" so that people
> |can get a text dump of the 0.9 LMDB databases. Let's see how it plays
> |out on Arch Linux and other rolling releases.
> |
> |Everything tracked
> |https://gitlab.com/bogofilter/bogofilter/-/work_items/21 because that's
> |easiest for me. We have a quick fix for 1.0 in place, but that does not
> |solve any migration pains yet, it's just "get it to work with 1.0".
> |
> |For FreeBSD, see
> |https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296519 and
> |https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296530
>
> Maybe the lmdb0 way of doing things is even better, i will do
> this, too.
> But this mdb_env_get_maxkeysize() thing you fixed and say
> "bogofilter bug" to in work_items/21 is a LMDB upstream bug:
> either they should fix their documentation, which still says
> _create() is what is expected, not _open(), or they need to fix
> their code.
> Do you talk with LMDB upstream already?

Steffen,

No I haven't talked to upstream, but if you find documentation and 
implementation to be inconsistent, please do - you have more detail 
knowledge than I do.


For me and/or bogofilter, it's a moot point: Even if it _were_ an 
upstream bug, what would we make of it? We need to fix it anyways. It 
doesn't hurt bogofilter to learn the max. key size a few lines later, 
mdb_env_create() may have been sufficient with a static max. key size 
and now it wants mdb_env_open() first because the max. key site is a 
property of the environment now. So what.

Look, the *API* change to 1.0 in itself required me to swap two function 
calls into LMDB, still works for 0.9, that makes the API breakage pretty 
smooth for what we want from the API, and I don't need #ifdefs or other 
ugly stuff.

The level of concern rises for the tasks "tell users to dump your 
databases to text before you update" and "tell distributors they need 
0.9 *AND* 1.0 so that users can transition their valuable databases". At 
least bogoutil -d and -l will need to work witrh both versions. So, 
let's discuss something useful for end users:

* Can I link both versions at least statically into one executable 
because the upstream has a portable way to rename its symbols?

* Can I trick a C++ compiler to import LMDB into a namespace so I can 
achieve that purpose without library support?

That would help quite a bit.


Matthias