Pierre Orhan

Do not hesitate to contact Pablo Diego or me if you have any questions about the work!
This work was done while I was at
@lsp-ens.bsky.social and finished at @institutducerveau.bsky.social !
Stay tuned for future works applying this approach to neurological recordings!

As a conclusion, artificial neural networks mirror some developmental steps and clarify sufficient conditions of their emergence. Yet models learn way too slowly: they require two orders of magnitude more words (tokens) than what children use to discover these linguistic structures!

The paper has some additional findings:
- Simpler linguistic structures (phonemes) are learned first, but semantic classes are not learned in a particular order.
- We invented a Topological Probe (enforcing topology rather than distance) to show the robustness of the approach.

Although the instantiation of the WordNet graph was strong in large models, it was small but significant in audio and small text models. We confirmed it with measurements based on semantic classes and visualized it through a nice coloring of the WordNet Graph, here for a 1B Pythia (text) model.

We found instantiation of these structures in an audio model: Wav2vec2 pretrained on English, but not when pretrained on other sounds!
By exploring these instantiations during pretraining, we evidence the emergence of each linguistic structure: phonemes emerge before lexical semantics and syntax.

To detect the instantiation of linguistic structures, we adapt the structural probe of Hewitt and Manning to semantics and phonetics. Semantic is given by the WordNet graph (hypernym relationships "is contained in"). Example: mammal is a hypernym of placental, itself a hypernym of carnivor, equine…

Findings:
- Linguistic structures are instantiated by a speech ANN.
- Phonemes are instantiated before semantics and syntax, mirroring the acquisition stages of children.
Methodological development:
- Probing of speech models through checkpoints
- Novel topological probe of WordNet

Novel study: Emergence of Phonemic, Syntactic, and Semantic Representations in Neural Networks.
Joint work with P. Diego supervision: Y. Lakretz, E. Chemla, Y. Boubenec, @jeanremiking.bsky.social
We explore the emergence of 3 linguistic structures in Neural Networks.
Arxiv: arxiv.org/abs/2601.18617

⚡️Check out our workshop tomorrow at @lpiparis.bsky.social, great speakers (@gabrielpeyre.bsky.social, @sdascoli.bsky.social, @samillingworth.com & many more) will cover Theory and Applications of Generative AI + Connexions with neuroscience 🧠
And there's food 🍰
➡️ genai-conference-website.vercel.app

Congratulations to Pierre Orhan (@pierreorhan.bsky.social) who successfully defended his PhD on "Learning dynamics in biological and artificial neural networks", conducted at @normalesup.bsky.social