Matthias Orlikowski

There’s plenty of evidence for political bias in LLMs, but very few evals reflect realistic LLM use cases — which is where bias actually matters.

IssueBench, our attempt to fix this, is accepted at TACL, and I will be at #EMNLP2025 next week to talk about it!

New results 🧵

My lab is looking for a Senior Scientist (= PostDoc with option of permanency)!

We are looking for someone interested in doing cutting-edge computational social science + helping us with data & software engineering 🤓.

See job ad for details jobs.uni-graz.at/en/jobs/7d14...

Planning on printing your #EMNLP2025 poster in Suzhou? There is an easy option available through the conference website: 2025.emnlp.org/poster-print...

“You speak Bavarian? Then you must be uneducated + closed-minded.”

🤯 Not your opinion? Good. But it might be your LLM’s!!

🧵 Check out our #EMNLP2025 paper, where we uncover concerning dialect bias in recent LLMs - including GPT-5.

#AI #Bias #Dialect #Fairness #LLM #NLProc #Safety

I have two fully funded PhD positions in my group. If you want to work on persona based dialogical models or using LLMs to fight online harms then consider applying!

Super excited to see the growing interest and awareness in diversity, variation, and pluralistic alignment in the NLP community! ✨ 🪩

Below 🧵 are some keynote talks and workshops that highlight this important direction 🤸

We’re incredibly proud to announce that our lab received two #ACL2025 awards: Best Social Impact Paper for @elisabassignana.bsky.social and @dirkhovy.bsky.social , and Outstanding Paper for @paul-rottger.bsky.social.

Congratulations to them and to all collaborators who made this possible!

Just arrived in Vienna for ACL 2025 🇦🇹 Excited to be here and to finally meet so many people in person!

We have several papers this year and many from @milanlp.bsky.social are around, come say hi!

Here are all the works I'm involved in ⤵️

#ACL2025 #ACL2025NLP

*pluralism* in human values & preferences (e.g. with personalisation) will also just
grow more important for a global diversity of users.

@morlikow.bsky.social is presenting our poster today at 1100. Also hyped for @michaelryan207.bsky.social's work and @verenarieser.bsky.social's keynote!

Very excited about all these papers on sociotechnical alignment & the societal impacts of AI at #ACL2025.

As is now tradition, I made some timetables to help me find my way around. Sharing here in case others find them useful too :) 🧵

📢 ACL 2025 this is a reminder of 🎤 #InvitedTalk by Verena Rieser from Google DeepMind "Whose Gold? Re-imagining Alignment for Truly Beneficial AI." on the technical & ethical challenges of building beneficial AI! 🤝 ⚖️ #ACL2025NLP #NLProc
Find out more: 2025.aclweb.org/program/keyn...

Looking forward to #acl2025 in Vienna! Excited to catch up. I am also hiring, one fully paid (100% E13) position for either a PhD student or a PostDoc in computational linguistics / digital humanities / corpus linguistics (earliest start date: October 2025). Who is interested?

Aaahhh I knew about this one but it didn’t show up with the keywords I was using on the accepted papers list! Thanks for reminding me 💖

[ACL'25] Subjectivity in Annotation Paper List Subjectivity in Annotation @ ACL 2025 Matthias Orlikowski What is this? Below you’ll find a list of papers that are relevant to subjectivity in annotation (human label variation, learning from disagre...

Here's a list of presentations at #acl2025 relevant to human label variation, learning from disagreement, perspectivism: docs.google.com/document/d/1...

Who's presenting on subjectivity in annotation (human label variation, learning from disagreement, perspectivism) at #ACL2025?

papers by e.g. @liweijiang.bsky.social @tiancheng.bsky.social @gabriellalapesa.bsky.social @romanklinger.de

keynote @verenarieser.bsky.social

link to full list below ⤵️

Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks LLM use in annotation is becoming widespread, and given LLMs' overall promising performance and speed, simply "reviewing" LLM annotations in interpretive tasks can be tempting. In subjective annotatio...

🗣️ Excited to share our new #ACL2025 Findings paper: “Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks” with Jad Kabbara and Deb Roy. Arxiv: arxiv.org/abs/2507.15821
Read about our findings ⤵️

More info on the paper in this thread: bsky.app/profile/morl...

Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also been used to label ...

I will present the poster on Monday July 28 during Poster Session 1 in the Human-Centered NLP track.

The session runs 11:00-12:30 in Hall 4 and 5, looking forward to discuss our work!

arxiv.org/abs/2502.20897

Picture of Matthias Orlikowski presenting a poster on the paper titled "Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions". The poster is similar to the one that will be presented at ACL 2025, showing a number of figures about the key results.

I will be at #acl2025 to present "Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions" ✨

Huge thank you to my collaborators Jiaxin Pei @paul-rottger.bsky.social Philipp Cimiano @davidjurgens.bsky.social @dirkhovy.bsky.social 🍰

more below

🎉 The @milanlp.bsky.social lab is excited to present 15 papers and 1 tutorial at #ACL2025 & workshops! Grateful to all our amazing collaborators, see everyone in Vienna! 🚀

I'll be presenting this tutorial in Vienna with co-organisers @daniel-russo.bsky.social, @helenabon.bsky.social, Yi-Ling Chung, @marcoguerini.bsky.social and invited civil society speakers Cathy Buerger (@dangerousspeech.org) and ‪@simofons.bsky.social‬ (Facta). Come along or join online!

🚨 New paper accepted at #ACL2025 Main conference! 🙌
"The AI Gap: How Socioeconomic Status Affects Language Technology Interactions"
w/ @amandacurry.bsky.social & @dirkhovy.bsky.social

Very happy for this first collab with @milanlp.bsky.social 🤝

🔗 lnkd.in/dtsH3gaX
🤗 lnkd.in/d7diuBJh

🧵👇
#ACL2025

Syllabus Syllabus: Computational Methods for Social and Political Science, Spring 2025 Recommended Reading: Christian, B., & Griffiths, T. (2016). Algorithms to live by: The computer science of human decisio...

Had the chance to revamp a PhD class on CSS. We're now using Colab notebooks to use Python for data analysis.

Check out the new syllabus (w/ reading, slides, background notebooks, and exercises plus solutions).

📚👇 bit.ly/438R6Hk

#ComputationalSocialScience #PhD

Content moderators are organizing against Big Tech Workers are fighting for working conditions and mental health support.

Incredible! "Content moderators who comb through harmful material uploaded to online platforms have formed a global trade union alliance...to improve working conditions. The Global Trade Union Alliance of Content Moderators (GTUACM) announced today in Nairobi"

www.theverge.com/news/658566/...

📢 Breaking changes to EMNLP submissions! If you plan to submit, pls make sure ALL co-authors know that:

- they have to register as reviewers by May 21st (can be done as soon as a paper is submitted)
- if they get assignments, it must be done on time and acc. to guidelines. Penalties now possible!

We (w/ @diyiyang.bsky.social, @zhuhao.me, & Bodhisattwa Prasad Majumder) are excited to present our #NAACL25 tutorial on Social Intelligence in the Age of LLMs!
It will highlight long-standing and emerging challenges of AI interacting w humans, society & the world.
⏰ May 3, 2:00pm-5:30pm Room Pecos

Happening now at #COMPTEXT2025 in a fantastic panel on "Advances in Misinformation, Toxicity & Hate Speech Detection" 👏
@pia-p.bsky.social presenting our latest work on how a more Perspectivist approach to NLP for algorithmic moderation could account for users’ differing views on what is harmful 🤔

Many thanks to @wired.com & @thiccreese.bsky.social for covering our work on stereotype bias!!
The new dataset is SHADES, now available here: huggingface.co/datasets/LanguageShades/BiasShades
The first of its kind for analyzing GenAI stereotyping across many languages+cultures.
Published at NAACL.🤗

📖For today's Reading Group @elisabassignana.bsky.social presented "Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates" by Aida Davani et al.

Paper: dl.acm.org/doi/10.1145/...

#NLProc

🎓For our last Friday's lab seminar, it was a pleasure to have
Pia Pachinger presenting "Modeling Toxicity Through the User’s Lens" and Luca Braghieri presenting "Article-level slant and polarization in news consumption on social media" ✨