I rewrote the entire write buffer, and journal backpressure system to be a more robust queue + messaging system. Also, now no more threads are started and stopped in the background on the fly. There is a single thread worker pool now (adjustable, at least 1), and that's it.

crates.io: Rust Package Registry

v3 pre.6 is out - probably the penultimate prerelease before going into release candidate(s). At this point, all major architectural reworks are done, and just some APIs need some changes. Followed by a final cleanup and stabilization phase.

crates.io/crates/fjall...

I didn't know what I was in for

And because less data is written to disk, it frees up availabe IOPS and increases SSD endurance.

With key-value separation, blobs are written at least twice (journal + blob file). So if a blob is compressible (let's say by 40%), we will still incur a write amp of 1.6x.

Journal compression uses more CPU, but because less data is transferred to OS and disk, it can actually be faster.

3.0.0 pre.5 is out and brings journal compression support.

Here's 16K JSON blobs being written:

Yesterday or so I ran a benchmark over night, and v3 scales much better for extremely large (100+ GB) databases

crates.io: Rust Package Registry

v3 pre-release 2 is out!

Still not file format -or API stable, but most importantly this release marks lsm-tree being *feature complete*.

crates.io/crates/fjall...

Well, it was a faulty RAM stick everybody

Thanks 😭

So I have now tried btrfs, xfs, ext4 on two diff. SSDs, updating to Ubuntu 24; and I'm still getting corruptions... at this point I don't really know what to do.

This is not code specific (RocksDB is also affected), yet I don't understand how my OS and all apps just seem to continue to work fine...

Without having numbers (right now), LMDB will probably read absurdly fast, while not so much space- or write efficient.
Most, if not all data, fits into RAM here. Would be interesting to benchmark 100s of GBs.

But I found redb to be slower (sometimes much slower) than LMDB, pretty much always.

And here's another read-heavy bench with 4K values, 5% sync random updates and 95% Zipfian reads.

ReDB uses 4x more disk space, and writes 6x slower. Interestingly, it's also ~5x slower in point reads - I think ReDB is not handling large values well; haven't read too much into its implementation.

Synchronous writes (fsync) I might add

95% Zipfian reads, 5% random updates on a Kingston PLP SSD with 16K JSON blobs - sled DNF! (OOM)

fjall uses LZ4 compression

constellation runs on a bare metal server with 4 dedicated CPU cores, 16GiB memory, 1TiB fast NVMe SSD attached, and 500mbit unmetered network connection.

**slightly slower in reading

*bench tool

sled (bloodstone) is not in the bench too rn, because you can't compile it next to sled w/ compression; plus it's super unstable - in general I found it to be faster in writing, smaller on disk, but similar or slightly slower in writing than sled 0.x...

Blob GC working again (now fully automatic) - much less impact on foreground work (old GC was stop-the-world)

Hopefully I will have results in 6 hours 👀

Noo 🥲

I wanted to expand it to a 1-hour run but it crashed because of a bug in the workload (I think), in the middle of the night.

Fun coloured bars and lines

Sponsor @fjall-rs on GitHub Sponsors Support fjall-rs's open source work

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!!

jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing!

bye

Sponsor @fjall-rs on GitHub Sponsors Support fjall-rs's open source work

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!!

jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing!

bye

Looks like the default fillfactor is 100%, which should be good for disk space. So it's probably dead row overhead and lack of page compression (?) plus maybe secondary indexes.

But yeah, it feels like things are slotting into place.

I have more than 50 git stashes and branches floating around... it's been a bit messy, but getting there.

I think I was lucky to fall into the pit of success because I think I made more progress the last two days than the last two months combined...

V3 pre-release(s) very soon... 👀

Add to that, sweeping API changes for better ergonomics that easily touch over a hundred files - Rust's doctests are a godsend here, I would have gone insane already without them

ngl v3 is a bit more involved than I originally planned or anticipated; I don't want breaking changes for a while, so it becomes this chase of making sure everything is really future-proof