Jelle Zuidema 🟥

Human individual judgements correlate even more strongly with the difference between a model's scores, but that says nothing about a model's abilities *in the wild*! This is contra Hu et al. '24 (www.pnas.org/doi/10.1073/...), & most importantly, provides a fresh dataset for use in this debate.

My favourite image from the paper, illustrating that LLMs are surprisingly weak at judging grammaticality. Human judgment correlates quite strongly with the *difference* in likelihood (or SLOR) that LLMs assign to pairs of grammatical & ungrammatical sentences, but that's the wrong measure.

#ACL2025

I'll be in Vienna only from tomorrow, but today my star PhD student Marianne is already presenting some of our work:

BLIMP-NL, in which we create a large new dataset for syntactic evaluation of Dutch LLMs, and learn a lot about dataset creation, LLM evaluation and grammatical abilities on the way.

BLiMP-NL: A Corpus of Dutch Minimal Pairs and Acceptability Judgments for Language Model Evaluation Abstract. We present a corpus of 8400 Dutch sentence pairs, intended primarily for the grammatical evaluation of language models. Each pair consists of a grammatical sentence and a minimally different...

Find our paper (w/ Michelle Suijkerbuijk, Zoë Prins, Jelle Zuidema @wzuidema.bsky.social, & Stefan Frank @stefanfrank.bsky.social), and dataset here:
📑 doi.org/10.1162/coli...
🗃️ doi.org/10.34973/tj4...

Here's a 2017 review paper by Monica Tamariz on all kinds of experiments with humans learning to agree on the meaning of signals. Much of it in the context of origins of language; some experiments use gestures to try to avoid biases from existing language.

www.annualreviews.org/content/jour...

Ik haal dat niet uit de tekst (de tekst leest als een onhandig compromis, maar wel één waar ik mee kan leven), ik zeg alleen dat het een uitkomst kan zijn van het onderzoek, als dat onderzoek met kennis van zaken wordt uitgevoerd.

Prima, maar dat is de klassieke discussie tussen fundi's en realo's. Maar dankzij de realo's hebben we in Europa, in veel landen èn aan veel universiteiten nu wel regels die een verantwoord gebruik van nieuwe technologie bevorderen.

En over de open brieven: Ik deel veel van de zorgen, maar ze zitten zo vol hyperbolen, en het advies "verban AI" is zo onrealistisch, dat ze gemakkelijk terzijde kunnen en zullen worden geschoven. Dus daar ben ik niet erg enthousiast over.

Zeker hebben werknemers die kans nu ook, maar tegelijk ontbreekt het bij veel medewerkers aan kennis over AI. Een uitkomst van zo'n studie kan ook zijn een beter beeld van wanneer AI wel verantwoordelijk kan worden ingezet, en wanneer niet, en vd mogelijkheden van open source & energiezuinige AI.

Mmm, je tirade richt zich op een subtext die ik er niet meteen in herken. Letterlijk staat er "een kans" en werknemers moeten "de kans krijgen", en daar lijkt mij niet veel mis mee.

Even voor mijn begrip: het is vanwege die twee zinnen in het akkoord, waarin in wat ambtelijke taal wordt opgeroepen tot 'een studie', dat je tegen gaat stemmen? Of is er op de rest ook wat aan te merken?

Het stuk is van the Editorial Board, dus zonder specifieke auteursnamen.

En heeft m.i. schokkend beperkt beeld over de internationale conflicten van de afgelopen decennia.

Interpretability Techniques for Speech Models — Tutorial @ Interspeech 2025

The @interspeech.bsky.social early registration deadline is coming up in a few days!

Want to learn how to analyze the inner workings of speech processing models? 🔍 Check out the programme for our tutorial:
interpretingdl.github.io/speech-inter... & sign up through the conference registration form!

Deadlines for PhD and Postdoc vacancies coming up: applications open until Monday June 2!

Ben ik heel cynisch als ik denk dat economische belangen bij de grote advocatenkantoren al heel lang belangrijker zijn dan "recht en gerechtigheid" en het beschermen van "kwetsbare groepen"? Eens, hoor, met de verontwaardiging en de oproep, maar volgens mij is alleen de "angst" nieuw.

Amsterdam Lectures in AI and Society ‹ clclab Melanie Mitchell, leading AI researcher from the Santa Fe Institute and key voice in the discussions on the abilities and inabilities of Large Language Models, will speak at Amsterdam Science Park.

This Friday at 3pm, Amsterdam's Computational Linguistics Seminar and the CLClab will host Melanie Mitchell, for a special edition of the Amsterdam Lectures in AI and Society. She will speak about "AI's Challenge of Understanding the World". Zoom link/ register:
clclab.netlify.app/2025/05/07/a...

Exciting to see a extensive study of -ity/-ness & frequency effects in LLMs! The same phenomena already inspired a beautiful analysis of pre-deep learning, Bayesian learning algorithms. Do you know Tim O'Donnell's papers & book?

Productivity and Reuse in Language www.jstor.org/stable/j.ctt...

I usually don't comment on my typos, but since we're talking AI innovation: isn't it infuriating that Google's gboard still hasn't learned that I never ever mean 'neutral network' when I swype 'neural network'? And that good-old-swypers need to choose between that and MS's even worse SwiftKey?

TIL that professor Kunihiko Fukushima, inventor of the neocognitron, is still an active researcher in his late 90s. The neocognitron is a convolutional neutral network (CNN); CNNs is turn were the model family with which deep learning revolution in Artificial Intelligence started. A living legend!

Isn't what you do in section 4 simply "representational similarity analysis"? I'm surprised not to see that term in the paper.

Congratulations on finding a nice opportunity to advertise your book. But if you want your hitpiece to be convincing you need better points than "he calls himself a historian but only studied... history" and "I found his secret donors on... his website". Don't waste your writing talent on nastiness.

Thanks. I mainly hope that the papers prove useful for people developing benchmarks and/or measures! For us, it was a hard paper to write, with so many differences in terms, writing styles, evaluation standards etc between psychology and NLP.

Good to see converging viewpoints!

Thanks for the reference to @hannawallach.bsky.social ++'s paper! So far only scanned it, but it looks like they arrive at
similar conclusions as we did in JAIR last year: odvanderwal.nl/2024/paper-c...

So, yes, I agree evaluation in NLP is a bit of a mess, & measurement theory has much to offer!

✨New paper ✨

Introducing 🌍MultiBLiMP 1.0: A Massively Multilingual Benchmark of Minimal Pairs for Subject-Verb Agreement, covering 101 languages!

We present over 125,000 minimal pairs and evaluate 17 LLMs, finding that support is still lacking for many languages.

🧵⬇️

Thanks for the reference! I have also wondered "What happened to mirror neurons?", so this paper looks like a useful overview. But what a missed opportunity to not have something like "Reflecting back on the mirror neuron debate" in the title. :).

Ah, good point - I did not check the rubric (not a reviewer for ICML).

Just for the record: reviewer 1, who gave us a 1 for not citing their favourite paper and for "not being super clear", may return from hell now, and instead spend some time in the purgatory.

Wow, grumpy lot those ICML reviewers!

Seems like a crazy way to spend these resources. I'm all for helping American scholars move to Europe, but those that already have an ERC grant waiting for them are pretty well off. The extra 1M€ would be better used to help *other* scholars, and not to make them dependent on those lucky colleagues.

Florence Nightingale: The pioneer statistician | Science Museum Discover how pioneering statistical methods helped prove that widespread reform of hospital care was vital.

I don't know if there are other examples directly from her; the one you linked is the one I have seen multiple times. E.g., here:
www.sciencemuseum.org.uk/objects-and-...