Lukas Warode

📣 New Preprint!
Have you ever wondered what the political content in LLM's training data is? What are the political opinions expressed? What is the proportion of left- vs right-leaning documents in the pre- and post-training data? Do they correlate with the political biases reflected in models?

The threat of analytic flexibility in using large language models to simulate human data: A call to attention Social scientists are now using large language models to create "silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research. How...

Can large language models stand in for human participants?
Many social scientists seem to think so, and are already using "silicon samples" in research.

One problem: depending on the analytic decisions made, you can basically get these samples to show any effect you want.

THREAD 🧵

We present our new preprint titled "Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation". We quantify LLM hacking risk through systematic replication of 37 diverse computational social science annotation tasks. For these tasks, we use a combined set of 2,361 realistic hypotheses that researchers might test using these annotations. Then, we collect 13 million LLM annotations across plausible LLM configurations. These annotations feed into 1.4 million regressions testing the hypotheses. For a hypothesis with no true effect (ground truth $p > 0.05$), different LLM configurations yield conflicting conclusions. Checkmarks indicate correct statistical conclusions matching ground truth; crosses indicate LLM hacking -- incorrect conclusions due to annotation errors. Across all experiments, LLM hacking occurs in 31-50\% of cases even with highly capable models. Since minor configuration changes can flip scientific conclusions, from correct to incorrect, LLM hacking can be exploited to present anything as statistically significant.

🚨 New paper alert 🚨 Using LLMs as data annotators, you can produce any scientific result you want. We call this **LLM Hacking**.

Paper: arxiv.org/pdf/2509.08825

Implications for political behaviour, communication, and representation are manifold, as 'left' and 'right' are central categories in polarised public discourse – which is particularly evident in pejorative usage, such as labelling political opponents as 'racist' or 'socialist'.

Both in- and out-ideological associations are externally validated by serving as seed words to scale parliamentary speeches. The resulting ideal points reflect party ideology across different specifications in the German Bundestag.

The mapping is based on associations from open-ended survey responses in German candidate surveys. Words are mapped into a semantic space using word embeddings and weighted by frequency. Construct validity is ensured by using alternative embeddings and frequency weightings.

Words associated with both left and the right are mapped to the semantic centre, where connotations can vary: 'freedom' has a positive connotation (it is primarily used by the respective in-group to describe left and the right), while 'politics' has a rather neutral connotation.

This framework yields associations that are driven by positive (in-ideology) and negative (out-ideology) associations. Examples: 'justice' (left) and 'patriotism' (right) are in-ideological associations; 'socialism' (left) and 'racism' (right) are out-ideological associations.

Left and right are essential poles in political discourse. We know little about how they are associated across the spectrum. I propose a 2-dimensional model that accounts for both semantics – is a term left or right – and position – are associations coming from the left or right.

My 2nd dissertation paper is out in @nature.com Humanities and Social Sciences Communications: www.nature.com/articles/s41...

I study and explore how associations with 'left' and 'right' vary systematically by semantic and political position.

Ja, die goldene Twitterzeit ist leider over

YouTube video by SamGordonRHK Sitcom Laugh Track

youtu.be/4VTBMznLrWs?...

📢 New Publication Alert!
Our (@msaeltzer.bsky.social)
latest article, "Issue congruence between candidates' Twitter communication and constituencies in an MMES: Migration as an exemplary case", has just been published in Parliamentary Affairs.
academic.oup.com/pa/advance-a...

Title and abstract of the paper.

Now out in Social Networks

Network analysis aspires to be “anticategorical,” yet its basic units—relationships—are usually readily categorized ('friendship,' 'love'). Thus, a nontrivial cultural typification is asserted in the very building blocks of most network analyses.

doi.org/10.1016/j.so...

a cartoon character from south park says " i m gonna need your help " ALT: a cartoon character from south park says " i m gonna need your help "

Calling all parliaments experts!
Say there's a debate in parliament, and a related vote. How frequently would these be on different days? different weeks? I don't mean different readings of bills, because these will also have different debates.
@sgparliaments.bsky.social #polisky #parlisky

Can banning political ideologies protect democracy? 🛡️🆚🗣️

Our (w. @valentimvicente.bsky.social) paper finds: punishing individuals might backfire. We study a West German policy banning "extreme left" individuals from working for the state.

#Democracy #PoliticalScience

🧵

url: osf.io/usqdb_v2

Happy to share my first published article based on my PhD in Party Politics with @journals.sagepub.com in open access!

doi.org/10.1177/1354...

Congrats, that is very interesting and great work! I work on something similar, but from a broader ideological perspective, in other words how ideological associations are asymmetrically used across the spectrum. Your work adds strong particular evidence for my overall argument, happy to see that!

The East in wolf’s clothing. Wolf attacks correlate with but do not cause far-right voting The resurgence of wolves in Germany has sparked intense debate, particularly in rural areas where wolf attacks on livestock are frequent. Prior resear…

Local reactions to wolf attacks in Germany are not powerful enough to influence the election decisions of many voters www.sciencedirect.com/science/arti...

Quite a strong final statement: "Descriptive research is important and it is a pity that the general obsession with causal estimates disincentivizes researchers from attempting to publish careful and detailed description."

2/2 In the second paper, which I will present in Barcelona @sgparliaments.bsky.social next week, @ortuttnauer.com and I investigate the dynamics of how and when parliamentary voting and speech-making align. I’m only hoping for good air conditioning, as I‘m sure about an amazing crowd again 🙂‍↔️

1/2 Another year, another @epsanet.bsky.social This year I had two papers accepted, but no visual proof that I presented joint work with Thomas Bräuninger on (the problems of) dynamic scale usage. I hope some nice people will capture 📸 me again next year in the Post-EPSA era @epssnet.bsky.social

2/2 In the second paper, which I will present in Barcelona @sgparliaments.bsky.social next week, @ortuttnauer.com and I investigate the dynamics of how and when parliamentary voting and speech-making align. I’m hoping for good air conditioning, as I’m sure the crowd will be amazing again!

So is it pronounced EPSS or EPSS? #epsa2025

In 1/2 an hour at #epsa2025, I’ll explain @lwarode.bsky.social and my approach to measuring divergence in gov-opp relations measurement based on parliamentary votes and speeches. The panel’s hidden at -1.A.05, so I hope you’ll find your way there (as will I)

Political expression of academics on Twitter - Nature Human Behaviour An analysis of nearly 100,000 academics on Twitter reveals strong progressive stances on climate and social issues, driven by a small, vocal subset. The study highlights potential gaps between academi...

A large-scale study of academics’ tweets shows disparities in who shapes academic discourse and how it reaches the public. @prashantgarg.bsky.social @trfetzer.com @imperialcollegeldn.bsky.social @uni-of-warwick.bsky.social
#AcademicTwitter #ClimateAction
www.nature.com/articles/s41...

How do elites view ideology? Analysing candidates’ associations of left and right Political ideology is a fundamental aspect of politics and a well-researched area of political science, but difficult to measure. By examining the sentiment of political elites’ associations with ‘...

www.tandfonline.com/doi/full/10....

🚨New publication @The_JOP on human biases in data annotation (w. Nora Webb Williams, @kevinaslett.bsky.social, John Wilkerson). Extremely important given the increasing societal reliance on AI tools often trained on human coders www.journals.uchicago.edu/doi/10.1086/...

A Tinder Test of Democratic Norms💋

New paper in @thejop.bsky.social with @bertous.bsky.social

We rely on a visual conjoint experiment, cross-sectional data, & panel data to show that affective polarization drives the normalisation of the far right among the centre-right 🇬🇧🇪🇸
doi.org/10.1086/736698

📣 New article alert 📣
Great insights shared by CDSS doctoral candidate Lukas Warode on elite attitudes towards political ideology. Check out the thread and his article for a deeper dive into the research area. Congrats @lwarode.bsky.social!👏