Jiatao Gu

World-consistent Video Diffusion with Explicit 3D Modeling Recent advancements in diffusion models have set new benchmarks in image and video generation, enabling realistic visual synthesis across single- and multi-frame contexts. However, these models still ...

Joint work by our awesome research intern Qihang Zhang, together with colleagues Shuangfei, Miguel, Kevin, Alex and Josh at Apple MLR!

Want to dive deeper? Check out our paper for full details

ArXiv: arxiv.org/abs/2412.01821
Project page: zqh0253.github.io/wvd/ (9/n, n=9)

WVD also supports controllable video generation. Given a single image, we estimate the 3D geometry via standard WVD inference, and project it to get partial XYZ images. Finally, WVD generates the RGB images jointly with the projected XYZ images through in-painting. (6/n)

For example, WVD can be directly applied to various single-image tasks. WVD can also take unposed images (video) as input, and infer XYZ images via “in-painting” strategy. With a post optimization procedure, the XYZ images can be converted to camera poses, and depth maps. (5/n)

At inference time, this joint distribution can be leveraged to estimate conditional distributions, such as P (XYZ | RGB) or P (RGB | XYZ). This capability makes WVD a foundation for supporting a wide range of downstream tasks. (4/n)

During training, WVD learns to generate 6D (RGB + XYZ) videos by modeling the joint probability P (RGB, XYZ), effectively capturing their interdependent structures and features. (3/n)

Existing multi-view/video diffusion model usually lack explicit 3D supervision (or guarantee), leading to potential 3D inconsistency and inefficient training.

In contrast, WVD models multi-view images, and explicit 3D geometry. Specifically, we represent the 3D geometry via XYZ images. (2/n)

WVD Pipeline

🤔Image-to-3D, monocular depth estimation, camera pose estimation, …, can we achieve all of this with just ONE model easily?

🚀Our answer is Yes -- Excited to introduce our latest work: World-consistent Video Diffusion (WVD) with Explicit 3D Modeling!

arxiv.org/abs/2412.01821

More interesting research work 🤔

Can anyone help add me to some starter pack🥲😰

I am seeking multiple PhD students passionate about Generative Intelligence and its applications in empowering AI agents to interact with the physical world to join us at UPenn CIS for the 2024-2025 academic cycle. You can find more information at www.cis.upenn.edu/graduate/pro...