Paul Lerner

🧑‍🔬I’m recruiting PhD students in Natural Language Processing @unileipzig.bsky.social Computer Science, together with @scadsai.bsky.social!

Topics include, but aren’t limited to:

🔎Linguistic Interpretability
🌍Multilingual Evaluation
📖Computational Typology

Please share!

#NLProc #NLP

The team meeting of the week was presented by Alexandre Vérine, from PSL, about "Quality and Diversity in generative models through the lens of f-divergences."
Thanks a lot for this interesting talk!

Accepted to a Workshop (1/2):

"Self-Retrieval from Distant Contexts for Document-Level Machine Translation", accepted to the Conference on Machine Translation (WMT25), from @ziqianpeng.bsky.social, @rachelbawden.bsky.social, @yvofr.bsky.social

There's many directions where this could go, multilingual, low-resource language, interpretability, depending on your profile, and the internship may lead to a PhD, provided we get funding!

Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs Paul Lerner, François Yvon. Proceedings of the 31st International Conference on Computational Linguistics. 2025.

As we found in aclanthology.org/2025.coling-... that BPE-based LLMs (i.e. pretty much all LLMs) did not handle prefixations well

Derivational morphology reveals analogical generalization in large language models | PNAS What mechanisms underlie linguistic generalization in large language models (LLMs)? This question has attracted considerable attention, with most s...

Basically the idea is to extend www.pnas.org/doi/10.1073/... to see how well LLMs model competition between affixes, not only suffixes (e.g. -ity vs. -ness) but also prefixes (e.g. un- vs. non-)

Come work with @yvofr.bsky.social @weissweiler.bsky.social and me at @mlia-isir.bsky.social for a M2 internship on Assessing the Morphological Competence of LLMs! For 5-6 months from February or March 2026. Paid 600€/month

What's the plural of "LLM-as-a-Judge"?

Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying o...

now on arxiv arxiv.org/abs/2510.20508

work done with @yvofr.bsky.social as part of the Democratic Commons programme, many thanks to our colleagues at Make, Sciences Po, and Sorbonne! about.make.org/democratic-c...

the dataset and code are available github.com/PaulLerner/2...

here's what one example of the dataset looks like, there are 72,234 just like this one (I regret my multimodal days where there were pictures in my papers)

We find that LLMs translate some political parties unfairly using a new version of EuroParl, fully multi-parallel and including (political) metadata
hal.science/hal-05328251

Google Colab

I tried for a pythonic library, have a look at the example notebook colab.research.google.com/github/PaulL...

🤔 ppllm is benchmarked against:
- a vllm-based implementation: 4.15 times faster!
- a naive hugging face implementation, which does not sort texts by length: 4.61 times faster!

🤔 ppllm implements windowed PPL, which allows to compute the PPL of arbitrarily long texts.
It aims to be feature complete for many information-theoretic metrics, including Perplexity (PPL), Surprisal, and bits per character (BPC), and their word-level counterparts.

GitHub - PaulLerner/ppllm: 🤔 A Python Library to Compute LLM's Perplexity and Surprisal 🤔 A Python Library to Compute LLM's Perplexity and Surprisal - PaulLerner/ppllm

introducing 🤔 ppllm, a Python Library to Compute LLM's Perplexity and Surprisal github.com/PaulLerner/p...

Contribute to current consultations - Comment l’IA peut-elle améliorer la vie des Français en limitant les risques ? - Make.org Finding proposals is easier when working together. Discover a democratic place where you can discuss the big issues you care about, submit your proposals concerning them and vote on proposals proposed...

make.org/FR/consultat...

"in 2025 we will have flying cars" 😂😂😂

Work done with Laurène Cave,
@haldaume3.bsky.social, Léo Labat, Gaël Lejeune, Pierre-Antoine Lequeu,
@bpiwowar.bsky.social, Nazanin Shafiabadi and @yvofr.bsky.social, read the paper here talnarchives.atala.org/ateliers/202...
Any feedback is appreciated :)

Last week, I presented my work on "Assessing the Political Biases of Multilingual LLMs" at the EALM workshop @ TALN 2025 ! Thanks again to the ANR Diké project for organizing the workshop

Investigating Length Issues in Document-level Machine Translation Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we chal...

📢 🎉 The team has one paper accepted to #MTsummit2025!
"Investigating Length Issues in Document-level Machine Translation" by @ziqianpeng.bsky.social, @rachelbawden.bsky.social and @yvofr.bsky.social in collaboration with @inriaparisnlp.bsky.social
📍 Geneva | 🗓️ 23-27,June
📕 arxiv.org/abs/2412.17592

"meticulously" is so absent from this list (from aclanthology.org/2025.coling-... )

Am I the only reviewer that actually fills this "Reviewer Checklist"? And why do Area Chairs never answer when the paper needs to be desk-rejected? And reviews are due in 3 days 🫠

For the EALM Workshop
"On Assessing the Political Biases of Multilingual Large Language Models" by @lernerp.bsky.social Laurène Cave, @haldaume3.bsky.social Léo Labat, Gaël Lejeune, Pierre-Antoine Lequeu, @bpiwowar.bsky.social Nazanin Shafiabadi and yvofr.bsky.social, collaborated with the STIH lab

PLMlatex, Éditeur LaTeX en ligne Un éditeur LaTeX en ligne facile à utiliser. Pas d’installation, collaboration en temps réel, gestion des versions, des centaines de modèles de documents LaTeX, et plus encore.

@mdlhx.bsky.social PS: I just found out we can actually share projects outside of CNRS plmlatex.math.cnrs.fr/6632958859wh...

Identifiant Un éditeur LaTeX en ligne facile à utiliser. Pas d’installation, collaboration en temps réel, gestion des versions, des centaines de modèles de documents LaTeX, et plus encore.

CNRS provides plmlatex.math.cnrs.fr that covers most of the features. I guess it's not so complicated to host (the software is open source)

", I am" 🤔
Don't you think this would increase the imbalance in multilingual LLMs?

Amazed at what a COLING paper could look like in the 80's

Hope you enjoyed our poster at #AISummit! I'm standing next to Pierre-Antoine Lequeu, @salimhafid.bsky.social, and @manonberriche.bsky.social but there's more people involved! Zoom-in to read their names or learn more about the project here about.make.org/democratic-c...