Moritz Hoffmann

Highlighting some of my team's recent work: We've changed Materialize to use swap instead of memory-mapped files, with nice performance and efficiency improvements.

Based on Timely 0.24, we released Differential Dataflow 0.17. Compatibility changes plus removing the Filter and Freeze wrappers because they weren't used and difficult to maintain. github.com/TimelyDatafl...

We also added a new way to chain multiple flat-map calls into a single Timely operator. Coming from Rust's iterator API, it's sometimes not obvious that Timely's API has a different cost structure (each transform renders an operator). Here, a builder can mitigate some of the cost.

I think these change are a great example of taking existing APIs and moving from row-first to batch-of-data. Computers generally seem better at handling data by column rather than row, and these changes allow Timely programs to represent data in their own form.

Distributor trait by antiguru · Pull Request #700 · TimelyDataflow/timely-dataflow Introduce a distributor that knows how to partition containers across multiple pushers. This moves the partition logic from container builders into a bespoke trait and implementation instead of mix...

A Distributor trait takes the concepts of Exchange, but moves away from row-by-row processing, opening up new kinds of containers that aren't row-first. For example, what is a row in a trie-structured container? The new trait allows to partition by whatever the user wants, not just by a "row".

Previously, the Container trait described behavior that is now separate: Accountable reveals a container's record count for progress tracking; nothing else. For higher-level APIs, IterContainer and DrainContainer reveal their contents (non-)destructively, for container-agnostic row-by-row operators.

Release timely-v0.24.0 · TimelyDataflow/timely-dataflow This version of Timely has some exciting new features. The Distributor trait offers a generalization of the Exchange type. It allows users to define custom distribution strategies for routing data...

We just released Timely Dataflow 0.24! Here are some exciting changes from @frankmcsherry.bsky.social and myself.
The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below.
github.com/TimelyDatafl...

GitHub - release-plz/release-plz: Publish Rust crates from CI with a Release PR. Publish Rust crates from CI with a Release PR. Contribute to release-plz/release-plz development by creating an account on GitHub.

github.com/release-plz/... for handling releases. GitHub CI for builds.

A lot of performance improvements went into this one!

The Materialize engineering team uncovered a rare concurrency bug 🪲in Rust’s 🦀 unbounded channels that could lead to double-free memory errors. After thorough debugging and working with the Rust and crossbeam communities, the fix is now part of @rust-lang.org 1.87.0.
🔗 bit.ly/3Fan1Om

Scaling queries on agent-produced data: How Delphi transformed its data infrastructure Join our webinar with Delphi to discover they evolved their data infrastructure to handle the rapid increase in AI-driven interactions with Materialize.

Agents generate more data and place more demand on systems than ever before—and standards like MCP will only accelerate this trend. Learn how Delphi is rethinking how they build data-intensive applications—from the db to the UI: bit.ly/42BZdf9

Materialize For Everyone: Introducing Self-Managed and our Free Community Edition Unlock real-time data transformation with Self-Managed Materialize. Now, deploy Materialize in your own cloud for security, compliance, and performance control—powered by the same battle-tested, incre...

I'm very excited to share that @materialize.com now has a Self-Managed option, as well as a Community Edition license that lets you play around with the exact same bits but for free.

materialize.com/blog/materia...

Jan '24: 1,338 kWh. That was a newly-built apartment in NYC. I don't miss this part!

For interested folks, the RustNYC presentation I gave about the `columnar` crate is now available in the repo: github.com/frankmcsherr...

It's not as good without the talk track, but it's got the benchmark results, you can admire the stylistic watercolors (so therapeutic to make; strong recommend).

We just landed the first use of the columnar crate in Materialize! Stay tuned for more.

github.com/MaterializeI...

Replica expiration: Limiting temporal filters' resource requirements Replica expiration is a new feature in Materialize that limits the resource requirements of temporal filters.

I wrote a bit about efficient maintenance of temporal filters in @materialize.com. By periodically restarting replicas, we can limit resource requirements (RAM and CPU), all without any other user-visible impact.

materialize.com/blog/replica...

An ocelot with an odd shape at the Museum für Naturkunde in Berlin.

I was recently coveted a 10 year journal on jetpens, because that would be cool to do, and realized that I already had one! The blog turns 10 this month, and I thought it might be worth taking a tour through each of the years and seeing what I can jot down about each.

github.com/frankmcsherr...

Origami tree at the @natural-history.bsky.social

Check out namespace.so

It'll certainly be an opt-in feature in its current form. Forcing a certain deployment pattern will be hard. Otoh, we could automate graceful replica restarts!

Sonne additional details by @frankmcsherry.bsky.social bsky.app/profile/fran...

So, we didn't solve the underlying problem, but mitigated it for many of our workloads.

Now, in Materialize time advanced about every second, and the result is high CPU utilization 😿 But: we restart most things every week, so updates beyond a week will never be surfaced! The mitigation we implemented is to drop such updates, and add some safeguards to never produce incorrect results. 🎉

Future updates are stashed at the input to indexes, and they'll only act on them once the time has advanced far enough. Because we use Differential Dataflows, the updates are only partially ordered. To extract ready updates, DD scans the data and splits it in ready and pending, ~ in linear time.

Improved performance for temporal filters Optimization time! We've significantly reduced the cost of using temporal filters in Materialize.

Sometimes it's easier to mitigate a problem than solve it: a feature I recently implemented reduces the memory and CPU @materialize.com requires to maintain future updates. Some details why solving the problem is hard and the mitigation easy-ish.
materialize.com/changelog/20...

Enjoying the call of the coquí in Puerto Rico

crates.io: Rust Package Registry

Just published lgalloc 0.4.0. The only change is that stats don't print errors anymore but forward to the caller. Docker on MacOS doesn't present numa information! crates.io/crates/lgalloc

Blended Wing Body Passenger Airplanes And The End Of Winged Tubes Ask someone to picture an airplane and they’re likely to think of what is essentially a tube with wings and a stabilizing tail tacked onto one end of said tube. Yet it is also no secret that …

Interesting article on the future of aircraft design: hackaday.com/2024/11/21/b...