Matthias Niessner

significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization.

🌍https://simongiebenhain.github.io/Pix2NPHM
🎥https://youtu.be/MgpEJC5p1Ts

Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen.

(1) large-scale registration of existing 3D head datasets, and
(2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals.
Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks

Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity.
Key to successful and generalized training of our ViT-based network are:

Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control.

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more… | Matthia... 📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive...

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢

We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME.

🌍https://peter-kocsis.github.io/IntrinsicImageFusion
🎥https://youtu.be/-Vs3tR1Xl7k

Great work by Peter Kocsis and Lukas Hollein!

3) optimize low-dimensional parameters for physically-grounded reconstructions.

The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset!

📢 Intrinsic Image Fusion for Multi-View 3D Material Reconstruction 📢

We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures

TUM AI Lecture Series - Building generative world models: progress and challenges (Ruiqi Gaoi) Abstract: Equipping AI models with the ability to imagine, reason, and act in the physical world is a crucial step toward achieving Artificial General Intelligence (AGI). Generative world models, whic...

Today in our TUM AI - Lecture Series we'll have the amazing Ruiqi Gao, Google DeepMind.

She'll talk about "𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐰𝐨𝐫𝐥𝐝 𝐦𝐨𝐝𝐞𝐥𝐬: progress and challenges".

Live stream: www.youtube.com/live/CkOSMqw...

7pm GMT+1 / 10am PST (Tue Dec 16th).

We also provide an interactive GUI to enable the exploration of our editing pipeline.

🌍 antoniooroz.github.io/PercHead/
📽️ youtu.be/4hFybgTk4kE

Great work by Antonio Oroz and and Tobias Kirschstein

by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt.

YouTube video by Matthias Niessner PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing

Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output.

In addition, our architecture seamlessly adapts to downstream tasks:

At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS.

📢📢 𝐏𝐞𝐫𝐜𝐇𝐞𝐚𝐝: 𝐏𝐞𝐫𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐇𝐞𝐚𝐝 𝐌𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐒𝐢𝐧𝐠𝐥𝐞-𝐈𝐦𝐚𝐠𝐞 𝟑𝐃 𝐇𝐞𝐚𝐝 𝐑𝐞𝐜𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 & 𝐄𝐝𝐢𝐭𝐢𝐧𝐠📢📢

PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text.

#ICCV last week was incredible — catching up with so many people, chatting about research, and, most importantly, having lots of fun.

Still hard to fathom this privilege as a researcher — getting to travel to such amazing places and be part of this brilliant community - Thanks!

The hot topic at #ICCV2025 was World Models.

They come in different flavors — (interactive) video models, neural simulators, reconstruction models, etc. — but the overarching goal is clear: Generative AI that predict and simulate how the real world works.

Hawaii on the same scale as the United Kingdom.

𝐺𝑒𝑛𝑒𝑟𝑎𝑡𝑜 𝑒𝑟𝑔𝑜 𝑠𝑢𝑚 — I generate, therefore I am.

GitHub - scannetpp/scannetpp: [ICCV 2023 Oral] ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes [ICCV 2023 Oral] ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes - scannetpp/scannetpp

for more documentation: github.com/scannetpp/sc...

Huge thanks to Yueh-Cheng Liu, as well as Chandan Yeshwanth and @niessner.bsky.social for their incredible work!

On the bright side, tooling for training has dramatically improved since then. Deep learning frameworks (PyTorch et. al) and scheduling systems such as SLURM or Kubernetes have become the backbone of modern AI.

Given the humongous compute demands of recent generative frontier AI models -- LLMs, image, and video models, etc. --, where compute is measured in Gigawatts, these challenges seem quite amusing.

The required compute was typically a couple of GPUs on a single desktop machine, trained over several days; e.g., AlexNet was trained on two GTX 580 3GB GPUs for 5-6 days.

In the 'early days' of modern deep learning (2012-2015) when ConvNets such as AlexNet or VGG came out, it was considered almost impractical to train an ImageNet classifier from scratch.

Fantastic retreat this weekend by our research groups!

Internal reviews, ideas brainstorming, paper reading, and much more! Of course also many social activities -- the highlight being our kayaking trip - lots of fun :)

All six of our submissions were accepted to #NeurIPS2025 🎉🥳

Awesome works about Gaussian Splatting Primitives, Lighting Estimation, Texturing, and much more GenAI :)

Great work by Peter Kocsis, Yujin Chen, Zhening Huang, Jiapeng Tang, Nicolas von Lützow, Jonathan Schmidt 🔥🔥🔥

We generate multiple videos along short, pre-defined trajectories that explore the scene in depth. Our scene memory conditions each video on the most relevant prior views while avoiding collisions.

Great work by Manuel Schneider & @LukasHollein

Can we use video diffusion to generate 3D scenes?

𝐖𝐨𝐫𝐥𝐝𝐄𝐱𝐩𝐥𝐨𝐫𝐞𝐫 (#SIGGRAPHAsia25) creates fully-navigable scenes via autoregressive video generation.

Text input -> 3DGS scene output & interactive rendering!

🌍http://mschneider456.github.io/world-explorer/
📽️https://youtu.be/N6NJsNyiv6I

We further propose a color-based densification and progressive training scheme for improved quality and faster convergence.

shivangi-aneja.github.io/projects/sca...
youtu.be/VyWkgsGdbkk

Great work by Shivangi Aneja, Sebastian Weiss, Irene Baeza Rojo, Prashanth Chandran, Gaspard Zoss, Derek Bradley

We operate on patch-based local expression features and increase the representation capacity by synthesizing 3D Gaussians dynamically by leveraging tiny scaffold MLPs conditioned on localized expressions.

ScaffoldAvatar: High-Fidelity Gaussian Avatars with Patch Expressions (#SIGGRAPH)

We reconstruct ultra-high fidelity photorealistic 3D avatars capable of generating realistic and high-quality animations including freckles and other fine facial details.

shivangi-aneja.github.io/projects/sca...