LMDB 1.0 <http://www.lmdb.tech/doc/>
# random
c
w
I have a hard time following their versioning - because their GH repo has master/master3 (which I think has encryption or something). They also have a variant of fossil and one of sqlite that uses LMDB instead of sqlite’s B-tree
I've also been checking out Meilisearch’s Rust wrapper (https://www.meilisearch.com/blog/3xfaster-vector-store ) - it seems to be the most maintained one
f
reminds me that I had experimented with a SQLite blob backend a while back. pushed it here. (caveats: haven't updated it in a while. includes benchmarks.)
LMDB was faster than unoptimized sqlite (for small blobs) and disk for large blobs.
The ability to allow configure different kinds of local stores could provide an upgrade path for LMDB 1.0 or something else if we don't want to be stuck on our LMDB fork. Probably worth me extracting that part of the branch.
w
I recall looking into these comparisons over the years (as it feels like the LMDB guy and drh had some competition going on back and forth), and it always sounded like the trade-offs ended up with +/- 10% on either side, with an overwhelming amount of the variance depending on the system.
I think one of the big problems I recall seeing was that a lot of the sqlite parts of the benchmarks weren’t actually fully optimized (e.g. missing stuff like the WAL and other craziness). Another example was incorrect page sizes
I looked into LMDB for something else I was working on, but I recall getting held up on some memory issue - where LMDB didn’t (doesn’t?) do memory compaction while transactions are in-flight, so I ended up using a ton of memory for a small project. So, I just moved back to old faithful sqlite, even if maybe it was a bit slower