Marc MacLeod

Thus there are only two postgres operations per message - initial insert, update on streaming complete. This should hopefully scale to the moon.

7. on llm stream complete, update the message in pg with the final message parts, set status to "completed", and close sse connections
8. the message part and status change is sync'd to client via zero - client now uses those parts to render the message instead of the previously streamed parts

5. token stream parts are kept in memory in the durable object, and streamed to connected clients over SSE
6. if a new client connects (or old client re-connects), it is immediately caught up to the present by streaming down the in memory parts on connect

4. we're opening an SSE connection per "streaming" message, not per thread. multiple messages can be streaming at the same time, within the same thread

3. on the client, when rendering any message that has status "streaming", open an SSE connection via ORPC to a durable object instance representing the chat thread

1. to kick off generating a reply, we insert a new message into postgres with status "streaming" and empty content
2. this new empty message is sync'd to the client via Zero

Streaming every token through pg / zero is not a great option at scale, so we're using SSE and cloudflare durable objects to handle the token streaming, and pg / zero for the durable persistence once streaming is complete.

Type-safe resumable streams and multi-user chat, courtesy of @Cloudflare durable objects, @zero__ms, and @orpc - how does it work?

Humble beginnings - every great product has to start somewhere

I can't wait to share more. Most importantly, I can't wait to start shipping again 🚀.

I'm not much of a sharer, but this go-around I'm committed to building more in public. Expect to see regular posts from me on things like ui / ux design, agentic patterns, running a lean company, and more in the weeds technical stuff.

It's too early to share specifics, I'm excited to work with our small team to re-imagine how we as humans interface with AI, each other, and the treasure trove of data that is fragmented across the apps we use every day.

Finally. It's time. A decade after starting my first company (wow time flies), I'm working on something new!

The new company is Datanaut, more below.

Love it! Where will the code live? Eager to take a look

Hello World