Rerun

✨ Massive Pipeline Refactor → One Framework for Ego + Exo Datasets, Visualized with @rerun.io 🚀

After a refactoring, my entire egocentric/exocentric pipeline is now modular. One codebase handles different sensor layouts and outputs a unified, multimodal timeseries file that you can open in Rerun.

MVP of Multiview Video → Camera parameters + 3D keypoints. Visualized with @rerun.io

Trying to wrap my head around fwd/bwd kinematics for imitation learning, so I built a fully‑differentiable kinematic hand skeleton in JAX and visualized it with @rerun.io new callback system in a Jupyter Notebook. This shows each joint angle and how it impacts the kinematic skeleton.

Exciting news for egui: there is a draft branch for switching the text handling to Parley, which will bring support for color emojis, right-to-left text, access to system fonts, and much more! github.com/emilk/egui/p...

@rerun.io v0.23 is finally out! 🎉 I’ve extended my @gradio-hf.bsky.social annotation pipeline to support multiview videos using the callback system introduced in 0.23.

Visualized with @rerun.io, I’ve integrated video‑based depth estimation into my robot‑training pipeline to make data collection as accessible as possible—without requiring specialized hardware.

I extended my previous @rerun.io and @gradio-hf.bsky.social annotation pipeline for multiple views. You can see how powerful this is when using Meta's Segment Anything and multi-view geometry. Only annotating 2 views, I can triangulate the other 6 views and get masks extremely quickly!

YouTube video by Parallel Domain Re: Can You Fool A Self Driving Car? Testing variations reveal what went wrong

Fun use of Rerun: www.youtube.com/watch?v=pxRG...

Here’s a sneak peek using @rerun.io and @gradio-hf.bsky.social for data annotation. It uses Video Depth Anything and Segment Anything 2 under the hood to generate segmentation masks and depth maps/point clouds. More to share next week.

Using @rerun.io , I established a baseline from the HoCAP dataset and conducted a qualitative comparison among the ground-truth calibrated cameras, Dust3r, and VGGT—all within rerun. The improvements are evident in both the camera parameters and the multi-view depth map/point cloud.

Rerun keeps growing! I’m so happy we started this thing, and that we’re building in the open, in Rust 🤘

11/ Want early access or excited about building the future of Physical AI? Reach out here 👉 5li7zhj98k8.typeform.com/to/a5XDpBkZ or check out open roles: rerun.io/careers. Let's build!

10/ It comes with visualization built-in for fast data observability over both online and offline systems. The query engine enables you to combine vector search and full dataframe queries, over both raw logs and structured datasets, to support robotics-aware data science and dataset curation.

9/ We are now building a new database and cloud data platform for Physical AI. The database is built around the same data model as the open-source project.

8/ This data model is core to our popular open source framework for logging and visualizing multimodal data, which companies like Meta, Google, @hf.co's LeRobot, and Unitree Robotics have adopted it in their own open-source projects.

7/ We spent the first 2 years of the company iterating on a data model for Physical AI that works for both messy online logs and efficient columnar storage of offline pipeline data.

6/ Rerun is building a unified, multimodal data stack—one data model and platform supporting both online and offline workflows seamlessly, retaining semantic richness throughout.

5/ Until now, no single tool has spanned the entire stack from online data capture, to offline dataset management.

4/ Hand-written online code is being replaced with ML, requiring advanced offline pipelines to collect, manage, label, and curate massive datasets.

3/ Physical AI systems rely on two key components:

Online systems: Run live on robots, processing data and interacting with the real world in real-time;
Offline systems: Run in data centers, analyzing data and improving online systems through training and simulations.

2/ Physical AI—robotics, drones, autonomous vehicles—is rapidly evolving, powered by advances in machine learning. But today's data stacks aren't built for this new era.

1/ We just raised $17M to build the multimodal data stack for Physical AI! 🚀

Lead: pointnine.com
With: costanoa.vc, Sunflower Capital,
@seedcamp.com
Angels including: @rauchg.blue, Eric Jang, Oliver Cameron, @wesmckinney.com , Nicolas Dessaigne, Arnav Bimbhet

Thesis: rerun.io/blog/physica...

More progress towards building a straightforward method to collect first-person (ego) and third-person (exo) data for robotic training in @rerun.io. I’ve been using the HO-cap dataset to establish a baseline, and here are some updates I’ve made (code at the end)

Finally finished porting mast3r-slam to @rerun.io and adding a @gradio-hf.bsky.social interface. Really cool to see it running on any video I throw at it, I've included the code at the end

I'm working towards an easy method to collect a combined third-person and first-person pose dataset starting Assembly101 from Meta, with near real-time performance via @rerun.io visualization. The end goal is robot imitation learning with Hugging Face LeRobot

Following up on my prompt depth anything post, I'm starting a bit of a miniseries where I'm going through the tutorials of
Lerobot to understand better how I can get a real robot to work on my custom dataset. Using @rerun.io to visualize
code: github.com/rerun-io/pi0...

Release 0.22.0 - Entity filter & improved partial update API · rerun-io/rerun Rerun is an easy-to-use visualization toolbox for multimodal and temporal data. Try it live at https://rerun.io/viewer. Python: pip install rerun-sdk Rust: cargo add rerun and cargo install rerun-...

Full release notes
github.com/rerun-io/rer...

The blueprint and stream panels support range selection using shift-click, which speeds up bulk operations like adding entities to views.

We've introduced a new panel for notifications. Instead of only having a short time to read a notification before it disappears, they are now stored until you dismiss them, so important errors and warnings no longer get lost.

We're also giving you more fine-grained time control.
You can now dynamically adjust the playback time, pause, and resume recordings via code. This means it is possible to synchronize the Viewer timeline with external tools. For now, this is for Web Viewer only (including Jupyter notebooks).