MaiNLP lab, LMU Munich

✨New paper✨

We find script (e.g. Cyrillic, Latin) to be a linear direction in the activation space of Whisper, enabling transliteration at test-time by adding such script directions to the activations — producing e.g. Cyrillic Japanese transcriptions.

VarDial @ EACL 2026, with important dates (see next post for text version). Photo CC-0.

VarDial 2026 will be colocated with @eaclmeeting.bsky.social! We're looking forward to your papers on NLP for similar languages, varieties and dialects :)

Deadline: Dec 19 (Jan 2 for pre-reviewed ARR papers)
sites.google.com/view/vardial...

Group photo at NeurIPS 2025 San Diego

Congrats to Pingjun, @beiduo.bsky.social , Siyao, Marie, and @barbaraplank.bsky.social for receiving the SAC Highlights reward!

Congrats to our team member Diego Frassinelli on the SAC Highlights award!

Awesome! We're also creating one currently and have included yours as a starter :)

Piper title ("A multi-dialectal dataset for German dialect ASR and dialect-to-standard speech translation") and a map of the German state Bavaria showing where the Franconian, Bavarian, and Alemannic dialect groups are spoken

At #Interspeech2025 I'm going to present Betthupferl, a dataset for German dialect ASR & dialect-to-standard speech translation! We analyze differences between dialectal & Standard German transcriptions, benchmark ASR models, and examine shortcomings of current ASR models & evaluation metrics.

UPDATE: Our poster presentation got moved to Tuesday, 16:00–17:30 (session 10)! #ACL2025NLP

Unsure which presentations to attend at #ACL2025? 🛎️🗣️

👥‪ @boleima.bsky.social Yuting Li, Wei Zhou, Ziwei Gong, @janetlauyeung.bsky.social Katja Jasinskaja @annefriedrich.bsky.social Julia Hirschberg, Frauke Kreuter @barbaraplank.bsky.social

👥‪ @boleima.bsky.social Berk Yoztyurk @carohaensch.bsky.social @xinpeng.bsky.social Markus Herklotz, Frauke Kreuter @barbaraplank.bsky.social @assenmacher.bsky.social

📝 Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter
🔎 263 languages, 10 similarity measures, 3 NLP tasks
👥 @verenablaschke.bsky.social Masha Fedzechkina @maartjeterhoeve.bsky.social
🔗 arxiv.org/abs/2501.14491
📁 Findings – long

📝Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?
🔎Analyzing how human-like LLMs are when taking reading, history, and economics tests
👥 @saeub.bsky.social , Diego Frassinelli, @barbaraplank.bsky.social
🔗 arxiv.org/abs/2506.09796
📁BEA workshop - Long

📝 GerMedIQ: A Resource for Simulated and Synthesized Anamnesis Interview Responses in German
🔎 We release a novel German anamnesis question-response dataset with human-simulated and LLM-augmented responses.
👥 @JHofenbitzer et al.
🔗 github.com/Jhofenbitzer...
📁SRW - Long

📝Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set
🔎Do LLMs encode and generalize discourse knowledge across languages?
👥 @florian-eichin.com @janetlauyeung.bsky.social @mhedderich.bsky.social @barbaraplank.bsky.social
🔗 arxiv.org/abs/2503.10515
📁Main - Long

📝LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
🔎We present a large-scale study of whether LLM judgments can be reliably used as proxies for human judgments
👥Anna Bavaresco et al.
🔗 arxiv.org/abs/2406.18403
📁Main - Short

📝 What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns
👥 @mhedderich.bsky.social Anyi Wang @raoyuan.bsky.social @florian-eichin.com Jonas Fischer @barbaraplank.bsky.social 

🔗 arxiv.org/abs/2504.158...

📁Main - Long

📝A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
👥 @beiduo.bsky.social Siyao Peng @annakorhonen.bsky.social @barbaraplank.bsky.social
🔗 arxiv.org/abs/2412.13942
📁ACL25 Findings-Long

📝Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models
🔎We study the relationship between circuits for highly compositional and functionally related tasks
👥@pmondorf.bsky.social Sondre Wold @barbaraplank.bsky.social
🔗 arxiv.org/abs/2410.01434
📁Main-Long

📝Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges
🔎We review existing datasets for evaluating LLMs’ pragmatic capabilities, outlining key challenges and promising future directions
🔗 arxiv.org/abs/2502.12378
📁Main - Long

📝Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study
🔎This study evaluates LLMs in generating German public opinions using open-ended survey data
🔗 arxiv.org/abs/2412.13169
📁Main - Long

Headed to ACL? MaiNLP & our most recent work will be there too👥📄
Come see what we’ve been working on!

📄 [ACL 2025 main] Circuit compositions: Exploring Modular Structures in Transformer-Based Language Models (doi.org/10.48550/arX...)

📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)

Correlations between transfer results per experiment (parsing, POS tagging, topic classification with different input representations) and similarity measures. The results vary a lot across experiments and measures – some are described in the next posts.

At #ACL2025NLP I'll present our analysis of the effect of linguistic similarity on cross-lingual transfer! We looked at how 10 similarity measures correlate w/ transfer results btwn 263 languages across 3 NLP tasks. Different similarity measures matter for diff. experiments (no one-size-fits-all)!

🤔 Can LLMs read between the lines?

Our another #ACL2025 paper surveys resources on how LLMs handle pragmatics like implicatures, deixis, and more. We map out a new landscape for both LLMs and linguistics in pragmatic research.

📄 arxiv.org/abs/2502.12378
🧠💬 #LLMs #Pragmatics

🚨 Can LLMs generate explanations that are as useful as human ones for modeling label distributions in NLI?🌹"A Rose by Any Other Name" shows that they can
💬 We explore scalable, explanation-based annotation via LLMs.
📍Come find us in Vienna 🇦🇹! (July 28, 18:00-19:30, Hall 4/5) #ACL2025NLP #acl2025

🎉 Our paper “Algorithmic Fidelity of LLMs in Generating Synthetic German Public Opinions” is accepted at #ACL2025 main conference as an oral presentation! 🇩🇪🤖

We study how well LLMs simulate real survey responses using open-ended German data, showing the left-leaning bias.

The future of AI literacy starts early! Our very own @fkoerner.bsky.social recently led a series of hands-on workshops teaching some of our youngest students yet 🌱🤖

Looking forward to my visit to Hamburg University and their Data Science group!