Soufiane Hayou

AISTATS 2026 will be in Morocco!

Shoutout to my collaborators Nikhil Ghosh and Bin Yu for their help with this project.

PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models Low-Rank Adaptation (LoRA) is a widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at a fraction of the cost of ful...

For more theoretical and empirical results, check our paper:
Paper: arxiv.org/abs/2506.20629
Code: github.com/soufiane001/...

✅ PLoP Consistently outperforms other strategies (Attn, MLP)
✅ Works across different post-training scenarios: supervised fine-tuning, reinforcement learning
✅ Minimal computational overhead
In the worst case, it ties with the best manual approach. Usually, it's better.

NFN measures the alignment between each module (in the pretrained model) and the finetuning task. NFN is a cheap metric that can be calculated in one forward pass. It is based on a large width analysis of module-data alignment and is well suited for LoRA finetuning.

Our solution: PLoP (Precise LoRA Placement) 🎯
Instead of guessing, it automatically identifies the optimal modules for LoRA placement based on a notion of module-data alignment that we call NFN (Normalised Feature Norms).

❌ Original LoRA paper: "Prioritize attention"
❌ Other papers: "Actually, put them in MLP"
❌ Everyone: just guessing and trying common target modules

LoRA is amazing for finetuning large models cheaply, but WHERE you place the adapters makes a huge difference. Most people are just guessing where to put them (Attention, MLP, etc).

Meet "PLoP" (Precise LoRA Placement) 🎯, our new method for automatic LoRA placement 🧵

The recent surge in available Research Scientist positions is correlated with the growing need for innovative approaches to scale and improve Large Language Models (LLMs). This trend is also driven by factors such as researchers leaving established companies for startups!

YouTube video by Rodrigo Amarante - Topic Tuyo (Narcos Theme) (Extended Version)

By far, the best intro song in the history of humankind

www.youtube.com/watch?v=zNPX...

Are we hitting a wall with AI scaling? 🤔

That "plateau" you're seeing in scaling law charts might not be a fundamental limit, but a sign of suboptimal scaling strategies! I wrote a blogpost about this:

www.soufianehayou.com/blog/plateau...

Speculative sampling accelerates inference in LLMs by drafting future tokens which are verified in parallel. With @vdebortoli.bsky.social , A. Galashov & @arthurgretton.bsky.social , we extend this approach to (continuous-space) diffusion models: arxiv.org/abs/2501.05370

People compare AI to past historic breakthroughs 🔄 (industrial revolution, internet, etc), but there's a crucial difference: In previous advancements, humans remained the most intelligent beings. This time, we're creating something that could surpass us 🤖. It's a singularity!⚡️