Reuben Bond

F3: The Open-Source Data File Format for the Future SIGMOD 2025

Our SIGMOD paper with our friends at Tsinghua + @wesmckinney.com + @pateljm.bsky.social on creating a next generation open-source data file format is out. F3 is a future-proof file format avoids the mistakes of Parquet.
📄 Paper: db.cs.cmu.edu/papers/2025/...
📁 Code: github.com/future-file-...

Why, hello there! 🪟

Thanks @rbn.bsky.social :D

I have an Aer one. You can get decent Tomtoc ones which mimic the designs of popular brands for less $ - unsure in the quality but I imagine they are decent.

Async/await was a developer experience regression compared with blocking APIs, but a huge advancement over the APIs it replaced, which were callback based.
I'm optimistic about virtual threads as a best of both worlds, if we can retain the benefits of futures & structured concurrency

Ah, thanks. How much of it is attributable to BEAM itself, I wonder. Using AOT instead of the JIT cuts memory usage in .NET by about half, for ex. IIRC, the numbers below are steady state under load, not startup, but either way

Thanks. Perhaps there is still low hanging fruit for elixir/phoenix/BEAM when it comes to reducing startup mem consumption. I'm sure it would still consume more than a Rust app given the nature of the runtimes

What is MB RES?

Pat Helland delivered a presentation on this paper: hpts.ws/papers/2024/...

In the last week I probably explained Rateless Set Reconciliation to a dozen other scientists. What an amazing paper and result, and already one year old.

It's just a toy / proof-of-concept, really, but it might be useful to refer to.

Reuben Bond - Orleans under the hood (Dotnetos Conference 2021) YouTube video by Dotnetos

Thank you, Jeremy! I gave a presentation about some of the design considerations in the serialization & RPC system: www.youtube.com/live/kgRag4E...

All things agents. I'm interested in builder communities rather than per-product servers

What's a good discord server for people building things in the AI/LLM space?

Committing the first value is safe as-is, but by playing more with this I believe subsequent Fast Rounds require additional safety rules:
1. Proposers only propose updates to known-committed values
2. Acceptors require the value's version (akin to slot number) has increased or values are identical

Did he have his onion juice with him or a couple of pocket eggs? If not, he's ngmi

gotta max out those T levels somehow

Feel free to dm

Is there a doc describing it?

Link?

"nothing more" is a bit too far: CRDTs have limitations on their behavior. You can't implement just anything using CRDTs. Their utility is quite limited and hence they are not widely deployed within datacenter based apps (I know of none, and no one has provided examples yet)

It depends on the CRDT, but that's not my area of expertise. The journaled grains use event sourcing (modified for geo distributed environments), and I believe you could implement CRDTs on top fairly easily, but the API also exposes coordination/sync operations for consistency.

What I described is not a CRDT. I am talking about ACID database transactions using optimistic concurrency control

You can version the data and check for conflicts at commit time, similar to a database transaction with optimistic concurrency control.

In the online case you can coordinate changes. In the offline case you cannot: you have divergent data replicas being updated independently. The changes need to be merged eventually and there is ambiguity as to how.

That's CRDT territory, but the goalposts just shifted

Interesting workshop and a lovely community, would very much recommend!

The sources can change but each one is authoritative (eg for fx rates) so you can assign a sequence number to each update which can be used to ensure consistency

One way would be to leave the import in a staging area until the user has completed their part of the workflow against a point in time snapshot. There is nothing you can do with a CRDT here which you cannot do without - there's no replication in this scenario

Why makes you say it's impossible without CRDTs? I don't see why it would be any more complex without CRDTs

Hi everyone! I'm a co-founder @dbos.dev, where we're building a serverless platform for highly reliable applications. I love conversations about databases, distributed systems, and anything technical. Thanks @qianli.dev for introducing me to Bluesky, and looking forward to meeting people here! 🦋