Pavel Tcholakov

I love this take on log-centric system architectures. It's been obvious for a while that this is a great way to model large-scale distributed systems; however practically it's been out of reach for many smaller teams. I am really excited for where we are heading!

Every System is a Log: Avoiding coordination in distributed applications Distributed coordination makes application complex and brittle. Because all systems eventually build on logs, we can use a shared-log approach to eliminate most coordination. We discuss this conceptua...

𝐄𝐯𝐞𝐫𝐲 𝐒𝐲𝐬𝐭𝐞𝐦 𝐢𝐬 𝐚 𝐋𝐨𝐠

Here is an idea on how to drastically 𝐫𝐞𝐝𝐮𝐜𝐞 𝐜𝐨𝐦𝐩𝐥𝐞𝐱𝐢𝐭𝐲 𝐚𝐧𝐝 𝐜𝐨𝐨𝐫𝐝𝐢𝐧𝐚𝐭𝐢𝐨𝐧 in distributed apps.

A fun way to look at this is similar to the idea of Turning the Database Inside Out - like Turning the Microservice Inside Out.

restate.dev/blog/every-s...

👇

So excited to announce the preview launch of S2! s2.dev/blog/intro

UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS 💀

Nice! We’re really enjoying WarpBuild; I’ve also had good results with Amazon CodeBuild but never used it as a GHA replacement.

Thrilled to be playing a part of this evolving story!

YouTube video by ChemicalBrothersVEVO The Chemical Brothers - Out Of Control (Official Music Video)

Got the perfect soundtrack youtu.be/6sOpbRL8R4g?...

The difficult part is that all the smart placement decisions happen within a zone, but you first need something at the regional API layer to decide upfront where to try send the request. Getting the balance wrong might mean much worse availability at the regional level during a single zone outage.

A primary driving factor is availability: the EC2 control plane wants to get out of the shared regional path and home onto a specific AZ as quickly as possible. This way you minimise shared points of failure and allow unaffected AZs to continue working in the presence of an outage.

I forget exactly what the classic networking API used to look like but making the AZ (or in the VPC world, subnet id) parameter optional. Seems easy enough but there were hard reasons why that was difficult to support!

Soaked up a lot of internal and external effort dealing with it. One of the worst consequences was that for large customers with multiple accounts, you wanted to know that AZ names are aligned within your organisation. But if you had launched any EC2 instances, even indirectly, we couldn’t change.

Fun fact, I was on the team that owned the service behind EC2 AZ randomisation a few (ok, quite a few…) years back. It was a very early, heavy handed decision to try better distribute placement - in hindsight should have been solved with better UI and APIs.

Important correction!

Is it really that weird? I thought it’s fairly common pattern in the Unix tools world. npm works the same way to separate its own arguments from those passed to scripts. Test binary is a bit of a leap, but I’d expect someone who understands how Rust programs get compiled to have an intuition for it.

Hi, friends! Nice to see this place pick up some buzz! Look forward to connect with the systems community over here.