DB backend support for lmdb?
Steffen Nurpmeso
steffen at sdaoden.eu
Wed Jul 18 01:56:57 CEST 2018
Matthias Andree wrote in <c3b69ec2-cc97-8a2b-4c0f-c67a1e6664e9 at gmx.de>:
|Am 17.07.2018 um 09:01 schrieb Matthias Andree:
|> Am 17.07.2018 um 01:35 schrieb Steffen Nurpmeso:
|>> Ahoi! And HA!!
|>>
|>> Steffen Nurpmeso wrote in <20180716221310.qDo5g%steffen at sdaoden.eu>:
|>>|Steffen Nurpmeso wrote in <20180715001400.pHS5O%steffen at sdaoden.eu>:
|>>||Matthias Andree wrote in <d38d0bd4-eb6e-6b0e-6252-6bf5223818f5 at an3e.de>\
|>>||:
|>>|||Am 21.06.2018 um 16:14 schrieb Steffen Nurpmeso:
|>>|||> Steffen Nurpmeso wrote in <20180529185111.0IjYp%steffen at sdaoden.eu>:
|>>|||>|Matthias Andree <matthias at an3e.de> wrote:
|>>|||>||Am 28.05.2018 um 23:57 schrieb Steffen Nurpmeso:
|>>|||> ...
|>>||Sorry for the long delay, this Thursday finally i have found time,
|>>| ...
|>>||causes some tests to fail. It is too late now, i have to go, and
|>>||revisit that on Monday.
|>> ...
|>>|So i post what i have, it is what i thought would be final.
|>>|I thought only, because we have five failing tests still.
|>>
|>> This has not changed. I will try on a GlibC and on FreeBSD
|>> tomorrow.
|> Hi Steffen,
|>
|> thanks for your contributions. I have tried on a plain Fedora 28 with
|> glibc, and also see five failed tests,
|>
|> FAIL: t.encoding
|> FAIL: t.nonascii.replace
|> FAIL: t.maint
|> FAIL: t.regtest
|> FAIL: t.upgrade.subnet.prefix
|>
|> Especially since the system tests already PASS, it seems we need to hunt
|> down a few borderline cases. I am using LMDB 0.9.22 for convenience.
|
|Hi Steffen,
|
|A bit of analysis on, for instance, t.nonascii.replace, reveals that one
|borderline case causing several of the failures shown above is that
|.WORDLIST_VERSION shows up with the LMDB driver where it does not show
|with the default BerkeleyDB or KyotoCabinet DB backends.
|I don't currently see why that is, and I am seriously wondering if the
|other drivers are correct... currently it's just correct by majority
|vote O:-)
You know, funnily i am staring at exactly the same since hours.
It helps and reduces the failures to three (mostly different) if
db_created() always returns false. Before that i had
bool
db_created(void *vhandle){
MDB_envinfo ei;
MDB_stat s;
struct a_bflm *bflmp;
bool created;
created = false;
if((bflmp = vhandle) != NULL){
/* no error defined */mdb_env_stat(bflmp->bflm_env, &s);
if(s.ms_entries == 0){
/* no error defined */mdb_env_info(bflmp->bflm_env, &ei);
if(ei.me_last_txnid == 0)
created = true;
}
}
return false;
return created;
}
|I am running with -Dvvvvxd to see the ds_read/ds_write calls.
Oh; i used a lot of
au Filetype c,cpp :iab <buffer> $P fprintf(stderr, "%s %u\n", __FILE__, __LINE__);
and something like
</dev/null BF_DEBUG_DB=1 BOGOFILTER_DIR=./ ../bogofilter -x abcdghilmrstuwyz -vvvv -C -d .
And i think from two to five, and from eleven to as of now, and
only running against walls. Really bitter, luckily pretty cheap.
The only other change i have is
-#define a_BFLM_MINSIZE (1u << 21)
+#define a_BFLM_MINSIZE 0x3FFFFFFFu/*(01u<<30)*//*(1u << 21)*/
which could be used to work around this what i still think is
a bug in LMDB, get rid of the complete replay log implementation
and simply call abort() whenever the MDB_MAP_FULL error happens.
This would surely improve performance, too.
|OTOH, I have identified the cause for bogofilter writing the
|.WORDLIST_VERSION and .ENCODING tokens in read-only databases.
|This has been fixed on the trunk, and I have synched the lmdb-support
|branch with those trunk changes.
|I have removed your workaround, and the self-test situation remains the
|same, with the same five failures for the LMDB branch, without
|introducing regressions for the KC, DB, SQLITE3 drivers.
My tests from yesterday were silly; even if i change t.config and
t.frame like e.g.
DB_TYPE=SQLite
export DB_TYPE
DB_EXT=db
export DB_EXT
BOGOFILTER="/usr/bin/bogofilter$EXE_EXT"
BOGOLEXER="/usr/bin/bogolexer$EXE_EXT"
BOGOTUNE="/usr/bin/bogotune$EXE_EXT"
BOGOUTIL="/usr/bin/bogoutil$EXE_EXT"
there still comes noise in that lets three tests fail (two for
DB).
|Before doing further development, please switch to
|^/branches/lmdb-support and/or update. I'm heading for bed now that
|r7064 has been committed.
Ok i will do this tomorrow. Great! Sourceforge now needs many
minutes to create a zip snapshot. I should boot FreeBSD and use
svn, that would be better maybe.
Anyway, i am somewhat out of ideas; let's see what the new svn
checkout will bring.
What do you think of the 1 GB initial size? Or even 2? It should
not hurt except for virtual size, at least optionally, with
a preprocessor switch?
|HTH a bit.
Let's see. Good Night, Matthias!
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
More information about the bogofilter-dev
mailing list