LMDB 1.0 / 0.9 database incompatibility

[Steffen Nurpmeso]

Matthias Andree via bogofilter-dev wrote in
 <c551c8de-02e6-4fee-9784-8143ad2f42b4 at gmx.de>:
 |Am 06.07.26 um 19:58 schrieb Steffen Nurpmeso:
 |> Matthias Andree via bogofilter-dev wrote in
 |> <d19429bc-83a0-4c32-8fa5-c6a88c0a9cec at gmx.de>:
 |>|Am 04.07.26 um 21:43 schrieb Steffen Nurpmeso:
 |>|> just saw 3cee1c59bc937 -- interesting! I have not tried LMDB 1.0
 |>|> yet, but saw some thread talking on it being entirely incompatible
 |>|> DB-wise. Ie, 0.9 databases are simply not supported at all!
 |>|> Whereas i understand Howard Chu a bit when he says
 |>|>
 |>|> The on-disk file format has changed in LMDB 1.0 and versions 0.9
 |>|> and 1.0 are mutually incompatible. You must use v0.9 #mdb_dump
 ...
 |>|> I think i myself follow certain advices and create completely new
 |>|> packages, ie keep bogofilter-lmdb and add bogofilter-lmdb1.
 |>|
 |>|For FreeBSD that would make sense for a transitional period; worst case
 |>|we need to make a copy of the ultimate LMDB 0.9.x into bogofilter's
 |>|sources and use it to build a feature-stripped "bogoutil" so that people
 |>|can get a text dump of the 0.9 LMDB databases. Let's see how it plays
 |>|out on Arch Linux and other rolling releases.
 |>|
 |>|Everything tracked
 |>|https://gitlab.com/bogofilter/bogofilter/-/work_items/21 because that's
 |>|easiest for me. We have a quick fix for 1.0 in place, but that does not
 |>|solve any migration pains yet, it's just "get it to work with 1.0".
 |>|
 |>|For FreeBSD, see
 |>|https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296519 and
 |>|https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296530
 ...
 |> But this mdb_env_get_maxkeysize() thing you fixed and say
 |> "bogofilter bug" to in work_items/21 is a LMDB upstream bug:
 |> either they should fix their documentation, which still says
 |> _create() is what is expected, not _open(), or they need to fix
 |> their code.
 |> Do you talk with LMDB upstream already?
 |
 |Steffen,
 |
 |No I haven't talked to upstream, but if you find documentation and 
 |implementation to be inconsistent, please do - you have more detail 
 |knowledge than I do.

Heaven no!  I have not looked into LMDB source ever since!  2018!
1.0 says

  #ifndef MDB_MAXKEYSIZE
  #define MDB_MAXKEYSIZE  0
  #endif

          /**     The maximum size of a key we can write to the environment. */
  #if MDB_MAXKEYSIZE
  #define ENV_MAXKEY(env) (MDB_MAXKEYSIZE)
  #else
  #define ENV_MAXKEY(env) ((env)->me_maxkey)
  #endif

  ..

  int ESECT
  mdb_env_get_maxkeysize(MDB_env *env)
  {
          return ENV_MAXKEY(env);
  }

So in how could the bogofilter commit fix anything, i really had
too look deeper.

 |For me and/or bogofilter, it's a moot point: Even if it _were_ an 
 |upstream bug, what would we make of it? We need to fix it anyways. It 

Yes, sure.

 |doesn't hurt bogofilter to learn the max. key size a few lines later, 
 |mdb_env_create() may have been sufficient with a static max. key size 
 |and now it wants mdb_env_open() first because the max. key site is a 
 |property of the environment now. So what.

Nothing really changed regarding this really i would think.
But like you say: that is not the point.

 |Look, the *API* change to 1.0 in itself required me to swap two function 
 |calls into LMDB, still works for 0.9, that makes the API breakage pretty 
 |smooth for what we want from the API, and I don't need #ifdefs or other 
 |ugly stuff.
 |
 |The level of concern rises for the tasks "tell users to dump your 
 |databases to text before you update" and "tell distributors they need 
 |0.9 *AND* 1.0 so that users can transition their valuable databases". At 
 |least bogoutil -d and -l will need to work witrh both versions. So, 
 |let's discuss something useful for end users:
 |
 |* Can I link both versions at least statically into one executable 
 |because the upstream has a portable way to rename its symbols?

I do not think they do.

 |* Can I trick a C++ compiler to import LMDB into a namespace so I can 
 |achieve that purpose without library support?

Neat trick.  'Could be this works if you include all the sources
etc of LDMB 0.9 in bogofilter.

Detection of the library that wrote the DB is impossible, i think
there is no file header or identification.
And as long as the loading fails, nothing embedded in the DB as
such can be used, now or in the future.

So what do you want to try?  If loading fails, or opening fails,
or what, then try once with the old embedded DB layer?
Then dump to a new DB, then atomically replace the old one?
To remark this may require quite some space and time.  My
bogofilter DB is almost 600 MB, and it was dumped/recreated just
last Saturday (monthly large backup + maintenance session)!)
And bogofilter can only ship one DB layer, if i recall correctly,
so the conversion has to be done?

Bad and sad situation.
Good night, Matthias, Ciao,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)