Nicolas Dufour

I'm commenting that number on slack with @nicolasdufour.bsky.social and I just realized that if you add the 16k active submissions at CVPR, even considering a sizeable overlap between the 2, there are currently well over 30k active papers in review.

That's nuts

Sadly i don't think DroPE will work for images / videos.
Both NoPE and DroPE rely on the causal mask to leak absolute PE. The number of tokens in the attention gets leaked because you can encode a bias that grows with the number of tokens.
So not a fix for images yet =(

It was a big pleasure to be in Nicolas's committee. Congratulations to Nicolas for the great work, and congratulations to the advisors too!

Apparently some people reported knowing of the bug before 11th of november so even before the release of the reviews

Yesterday, @nicolasdufour.bsky.social defended is PhD. I really enjoyed the years of collaboration w/ @vickykalogeiton.bsky.social (& @loicland.bsky.social)

Video: youtube.com/live/DXQ7FZA...

Big thanks to the jury @dlarlus.bsky.social @ptrkprz.bsky.social @gtolias.bsky.social A. Efros & T. Karras

Congrats Nicolas ! On the PhD and on those beautifully crafted slides 🤩

Nicolas ( @nicolasdufour.bsky.social ) is defending his PhD right now.

I was so in awe of the presentation that I even forgot to take pictures 😅

Yes it's latent space just because i had my setup that way. Might try in pixel space in the future.

Yes it's the raw prediction, we predict the velocity directly

It's also very domain dependent. I know that for example, x-pred works better than epsilon pred for human motion generation.

Epsilon loss was used for a while for image generation since DDPM.
Recently it was more flow matching (or v-loss) that is mostly used since SD3 basically.
From my experience, flow doesn't really improve quality, but sampling in fewer steps works better than epsilon prediction

Don't drop your samples! Coherence-aware training benefits Conditional diffusion Conditional diffusion models are powerful generative models that can leverage various types of conditional information, such as class labels, segmentation masks, or text captions. However, in many rea...

Thanks for the pointer! We were doing something similar in "Don't drop your samples" (arxiv.org/abs/2405.20324)

MIRO is quite different in the sense we focus on improving pretraining (not finetuning). Also, we explore the advantages of having multiple rewards to push the Pareto frontier.

Yes, thanks for pointing it out, will try to clarify

Check out our new work: MIRO

No more post-training alignment!
We integrate human alignment right from the start, during pretraining!

Results:
✨ 19x faster convergence ⚡
✨ 370x less compute 💻

🔗 Explore the project: nicolas-dufour.github.io/miro/

Image generation becomes much more energy efficient. 👍

I'm super happy about Nicolas' latest work, probably the magnum opus of his PhD.

Read the thread for all the great details.
The main conclusion I draw from this work is that better pretraining, in particular by conditioning on better data, allows us to train SOTA models at a fraction of the cost.

Work with @lucasdegeorge.bsky.social @arrijitghosh.bsky.social @vickykalogeiton.bsky.social and @davidpicard.bsky.social.

This will be the last work of my PhD as I will be defending the 26th of November!

MIRO: Multi-Reward Conditioning for Efficient Text-to-Image Generation Train once, align many rewards. MIRO achieves 19× faster convergence and 370× less compute than FLUX while reaching GenEval score of 75. Controllable trade-offs at inference time.

MIRO demonstrates that aligning T2I models during pretraining is not only viable but superior: it's faster, more compute-efficient, and provides fine-grained, interpretable control.

Project page for all the details: nicolas-dufour.github.io/miro

The explicit reward conditioning allows for flexible trade-offs, like optimizing for GenEval by reducing the aesthetic weight in the prompt. We can also isolate the look of a specific reward or interpolate them via multi-reward classifier-free guidance

MIRO excels on challenging compositional tasks (Geneval here)

The multi-reward conditioning fosters better understanding of complex spatial relationships and object interactions.

Despite being a compact model (0.36B parameters), MIRO achieves state-of-the-art results:

GenEval score of 75, outperforming the 12B FLUX-dev (67) for 370x less inference cost.
Conditioning on rich reward signals is a highly effective way to achieve large model capabilities in a compact form!

MIRO dramatically improves sample efficiency for test-time scaling.

On PickScore, MIRO needs just 4 samples to match the baseline's 128 samples (a 32x efficiency gain).
For ImageReward, it's a 16x efficiency gain

This demonstrates superior inference-time efficiency for high-quality generation.

Traditional single-objective optimization often leads to reward hacking. MIRO's multi-dimensional conditioning naturally prevents this by requiring the model to balance multiple objectives simultaneously. This produces balanced, robust performance across all metrics contrary to single rewards.

The multi-reward conditioning provides a dense supervisory signal, accelerating convergence dramatically. A snapshot of the speed-up:

AestheticScore: 19.1x faster to reach baseline quality.
HPSv2: 6.2x faster.

You can clearly see the improvements visually

This reward vector s becomes an explicit, interpretable control input at inference time. We extend classifier-free guidance to the multi-reward setting, allowing users to steer generation toward jointly high-reward regions by defining positive (s^+) and negative (s^−) targets.

MIRO trains p(x∣c,s) by conditioning the generative model on a vector s of reward scores for each image-text pair. Instead of correcting a pre-trained model, we teach it how to trade off multiple rewards from the start.

We introduce MIRO: a new paradigm for T2I model alignment integrating reward conditioning into pretraining, eliminating the need for separate fine-tuning/RL stages. This single-stage approach offers unprecedented efficiency and control.

- 19x faster convergence ⚡
- 370x less FLOPS than FLUX-dev 📉

Kickstarting our workshop on Flow matching and Diffusion with a talk by Eric Vanden Eijnden on how to optimize learning and sampling in Stochastic Interpolants!

Broadcast available at gdr-iasis.cnrs.fr/reunions/mod...

Final note: I'm (we're) tempted to organize a challenge on that topic as a workshop at a CV conf. ImageNet is the only source of images allowed and then you compete to get the bold numbers.

Do you think there would be people in for that? Do you think it would make for a nice competition?

Very proud of our recent work, kudos to the team! Read @davidpicard.bsky.social’s excellent post for more details or the paper arxiv.org/pdf/2502.21318