Rohan Desai

This will be the foundation for being able to branch @kafkastreams.bsky.social apps to support seamless blue/green deploys. It also allows you to branch a previous version of the state to debug an issue from the past.

Powerful stuff coming to the world of stream processing!

Q&A: Scaling Event Driven Architectures with Metronome | LinkedIn We're hosting a live Q&A session with two engineers who have seen it all: - Casey Crites, a founding engineer at Metronome, implemented and scaled the Kafka pipeline at Metronome from zero to where i...

For any Kafka Streams fans out there who want to know what it's like scaling an EOS application up to multiple TB of state at ~10k+ records/s with minimal downtime -- make sure to catch the Q&A we're hosting tomorrow with Metronome.

Join us Dec 12th at 9:30am PST!

www.linkedin.com/events/72647...

This was a fun conversation with @thegeeknarrator.bsky.social and @yingjunwu.bsky.social about the world of SQL stream processing on one side and event driven apps on the other. Thanks for having me on the show, Kaivalya!

It's said that Silicon Valley is special because the density of smart and motivated people leads to chance encounters that don't happen elsewhere.

I can attest to that.

Here's how a coffee resulted in @responsive.dev building a database optimized for stream processing in 8 months. (1/n)

From one of @responsive.dev 's customers:

"will report back .. when we start to change ttl lengths, which we're currently set up to do through launchdarkly based on the client and environment...pretty slick."

They are changing the TTL of rows in their Kafka Streams state stores dynamically 🤩

Welcome @ableegoldman.bsky.social to BlueSky! Kafka Streams extraordinaire!

It makes a huge difference because it lets you avoid doing remote lookups for the first key in each window. Without the filters we were seeing a 50% drop-off in throughput at the start of each window.

Some problems are impossible to solve without stream processing: for instance, did you know that metronome.com leverages Kafka and Kafka Streams to deliver real time billing features like spend limits at scale? (1/3)

Coming from Kafka land, I’m always surprised at how “time” is an afterthought in many other databases.

Hoping SlateDB gets it right: 1) have only one clock, 2) let users specify the clock, 3) enforce monotonic clocks, 4) use seq numbers (not time) for txns.

Are we missing anything?

Looking forward to this! We just started using SlateDB in @tensorlake.bsky.social’s compute engine, looking forward to not needing dynamodb for CAS. One less moving part!

@chris.blue when is the new release coming? :)

object_store: Conditional put and rename_if_not_exist on S3 · Issue #6285 · apache/arrow-rs Is your feature request related to a problem or challenge? Please describe what you are trying to do. This is now supported on S3: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-condi...

And it's done. Thank you so much @benesch.bsky.social for doing this. Now SlateDB will be able to have CAS for all major object stores. 🔥 It just works.

SlateDB: 'Bottomless' Databases Built on Cloud Object Stores SlateDB can dramatically cut costs of running a key-value store in the cloud, as long as users don't mind a bit of latency.

The new stack just put up a post on SlateDB!

👋 @vigneshc.bsky.social! All, Vinesh is a maintainer and committer for slatedb.io. He implemented the manifest and has been working on fizzbee.io proofs with JP. In his spare time, he’s a manager for Azure’s streaming team. 😁

I am soon interviewing @apurvamehta.com from @responsive.dev and Yingjun Wu from @risingwave.bsky.social on all things stream processing.

This is going to be an amazing episode. Feel free to ask any questions related to Stream processing and I will add the most interesting ones. Shoot!

Yep flamegraph-rs is just for CPU profiles. I guess I should have figured given you mentioned bytehound/valgrind 😀. Bytehound has worked great for me in the past (on linux).

For a cpu profile, I found flamegraph-rs (github.com/flamegraph-r...) to be a useful starting point. I've only tried on linux but their README says it works on osx using dtrace. The README also suggests trying github.com/mstange/samply. It says it has better osx support. Not sure what that means.

If not RocksDB, then what? Why SlateDB is the right choice for Stream Processing. Why we think SlateDB is the right foundation for a state storage service for Kafka Streams.

Is it end of the road for RocksDB in stream processing?

Disaggregated state is the clearly superior architecture, with @responsive.dev investing heavily in SlateDB.io while Flink 2.0 has forked RocksDB.

Here's why we've bet on SlateDB for Kafka Streams: www.responsive.dev/blog/why-sla...

RFC for supporting checkpoints/clones by rodesai · Pull Request #278 · slatedb/slatedb A cloud native embedded storage engine built on object storage. - RFC for supporting checkpoints/clones by rodesai · Pull Request #278 · slatedb/slatedb

This is gonna be cool...

"The writer will also support initializing a new database from an existing checkpoint. This allows users to 'fork' an instance of SlateDb, allowing it to access all the original db’s data from the checkpoint, but isolate writes to a new db instance."