@bkhmsi.bsky.social
PhD at EPFL 🧠💻 Ex @MetaAI, @SonyAI, @Microsoft Egyptian 🇪🇬
Looking forward to be speaking at IndabaX Sudan on Building Responsible and Ethical LLMs!
📅 Saturday, December 13th
⏰ 2:00 PM (GMT+2)
Register here: docs.google.com/forms/d/e/1F...
See you all there! :)
Not attending NeurIPS this year, but very much looking to connect.
I’m seeking a PhD research internship next summer in AI for Science, especially where AI meets brain and cognitive sciences. 🧠
If you’re hiring, I’d love to connect!
bkhmsi.github.io
I finally found time to update the Egyptians in AI Research website, apologies for the delay!
Super excited to share that we now feature 227 incredible Egyptian researchers!! 🤯
Link: bkhmsi.github.io/egyptians-in...
You can learn more about our work here: language-to-cognition.epfl.ch
Thanks to all my co-authors @gretatuckute.bsky.social, @davidtyt.bsky.social, @neurotaha.bsky.social and my advisors @abosselut.bsky.social and @mschrimpf.bsky.social!
On my way to #EMNLP2025 🇨🇳
I’ll be presenting our work (Oral) on Nov 5, Special Theme session, Room A106-107 at 14:30.
Let’s talk brains 🧠, machines 🤖, and everything in between :D
Looking forward to all the amazing discussions!
10/
🙏 Huge thanks to my incredible co-authors @cndesabbata.bsky.social, @gretatuckute.bsky.social, @eric-zemingchen.bsky.social
and my advisors @mschrimpf.bsky.social and @abosselut.bsky.social!
9/
🔗 Explore MiCRo:
🌐 Website: cognitive-reasoners.epfl.ch
📄 Paper: arxiv.org/abs/2506.13331
🤗 HF Space (interactive): huggingface.co/spaces/bkhmsi/cognitive-reasoners
🧠 HF Models: huggingface.co/collections/bkhmsi/mixture-of-cognitive-reasoners-684709a0f9cdd7fa180f6678
8/
We now have a collection of 10 MiCRo models on HF that you can try out yourself!
🧠 HF Models: huggingface.co/collections/bkhmsi/mixture-of-cognitive-reasoners-684709a0f9cdd7fa180f6678
7/
We built an interactive HF Space where you can see how MiCRo routes tokens across specialized experts for any prompt, and even toggle experts on/off to see how behavior changes.
🤗 Try it here: huggingface.co/spaces/bkhms...
(Check the example prompts to get started!)
6/
We also wondered: if neuroscientists use functional localizers to map networks in the brain, could we do the same for MiCRo’s experts?
The answer: yes! The very same localizers successfully recovered the corresponding expert modules in our models!
5/
One result I was particularly excited about is the emergent hierarchy we found across MiCRo layers:
🔺Earlier layers route tokens to Language experts.
🔻Deeper layers shift toward domain-relevant experts.
This emergent hierarchy mirrors patterns observed in the human brain 🧠
4/
We find that MiCRo matches or outperforms baselines on reasoning tasks (e.g., GSM8K, BBH) and aligns better with human behavior (CogBench), while maintaining interpretability!!
3/
✨ Why it matters:
MiCRo bridges AI and neuroscience:
🤖 ML side: Modular architectures make LLMs more interpretable and controllable.
🧠 Cognitive side: Provides a testbed for probing how the relative contributions of different brain networks support complex behavior.
2/
🧩 Recap:
MiCRo takes a pretrained language model and post-trains it to develop distinct, brain-inspired modules aligned with four cognitive networks:
🗣️ Language
🔢 Logic / Multiple Demand
🧍♂️ Social / Theory of Mind
🌍 World / Default Mode Network
🚀 Excited to share a major update to our “Mixture of Cognitive Reasoners” (MiCRo) paper!
We ask: What benefits can we unlock by designing language models whose inner structure mirrors the brain’s functional specialization?
More below 🧠👇
cognitive-reasoners.epfl.ch
Excited to be part of this cool work led by Melika Honarmand!
We show that by selectively targeting VLM units that mirror the brain’s visual word form area, models develop dyslexic-like reading impairments, while leaving other abilities intact!! 🧠🤖
Details in the 🧵👇
Huge thanks to my amazing collaborators: @gretatuckute.bsky.social, @davidtyt.bsky.social, @neurotaha.bsky.social & advisors @abosselut.bsky.social and @mschrimpf.bsky.social!
You can find more about our paper on the project's website: language-to-cognition.epfl.ch
Paper: arxiv.org/abs/2503.01830
Now that the ICLR deadline is behind us, happy to share that From Language to Cognition has been accepted as an Oral at #EMNLP2025! 🎉
Looking forward to seeing many of you in Suzhou 🇨🇳
1/🚨 New preprint
How do #LLMs’ inner features change as they train? Using #crosscoders + a new causal metric, we map when features appear, strengthen, or fade across checkpoints—opening a new lens on training dynamics beyond loss curves & benchmarks.
#interpretability
NEW PAPER ALERT: Recent studies have shown that LLMs often lack robustness to distribution shifts in their reasoning. Our paper proposes a new method, AbstRaL, to augment LLMs’ reasoning robustness, by promoting their abstract thinking with granular reinforcement learning.
23.06.2025 14:32 — 👍 7 🔁 3 💬 1 📌 1Check out @bkhmsi.bsky.social 's great work on mixture-of-expert models that are specialized to represent the behavior of known brain networks.
18.06.2025 10:46 — 👍 3 🔁 1 💬 0 📌 0
11/ 🌐 Links
Paper: arxiv.org/abs/2506.13331
Project Page: bkhmsi.github.io/mixture-of-c...
Code: github.com/bkhmsi/mixtu...
Models: huggingface.co/collections/...
In collaboration with: @cndesabbata.bsky.social, @eric-zemingchen.bsky.social, @mschrimpf.bsky.social, & @abosselut.bsky.social
10/ 🧾 Conclusion:
MiCRo weaves together modularity, interpretability & brain-inspired design to build controllable and high-performing models, moving toward truly cognitively grounded LMs.
9/ 💡 Key insights:
1. Minimal data (~3k samples) in Stage 1 can induce lasting specialization
2. Modular structure enables interpretability, control, and scalability (e.g., top‑2 routing can boost performance)
3. Approach generalizes across domains & base models
8/ 🧬 Brain alignment:
Neuroscience localizers (e.g., for language, multiple-demand) rediscover the corresponding experts in MiCRo, showing functional alignment with brain networks. However, ToM localizer fail to identify the social expert.
Figures for MiCRo-Llama & MiCRo-OLMo.
7/ 🧩 Steering & controllability:
Removing or emphasizing specific experts steers model behavior: Ablating logic expert hurts math accuracy; suppressing social reasoning improves math slightly—showcasing fine-grained control.
6/ 🔄 Interpretable routing:
Early layers route most tokens to the language expert; deeper layers route to domain-relevant experts (e.g., logic expert for math), matching task semantics.
5/ 📈 Performance gains:
We evaluate on 6 reasoning benchmarks (MATH, GSM8K, MMLU, BBH…), MiCRo outperforms both dense and “general‑expert” baselines: modular models with random specialist assignment in Stage 1.
4/ 📚 Training curriculum (3 stages):
• Stage 1: Expert training on small curated domain-specific datasets (~3k samples)
• Stage 2: Router training, experts frozen
• Stage 3: End-to-end finetuning on large instruction corpus (939k samples)
This seeds specialization effectively.
3/ ⚙️ Architecture:
We start with a pretrained model (e.g. Llama‑3.2‑1B). Clone each layer into four experts. Then, a light router assigns tokens dynamically to a single expert (top‑1 routing) per layer. Keeping a comparable number of active parameters to the base model.
