Carolina Haensch

Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study In recent research, large language models (LLMs) have been increasingly used to investigate public opinions. This study investigates the algorithmic fidelity of LLMs, i.e., the ability to replicate th...

📄 arxiv.org/abs/2412.13169
🗓️ Monday, July 28, 14:00–15:30
📍Room 1.62, Session 3: IP-Orals

Looking forward to seeing you in Vienna #ACL2025! 👋
#LLMs #PublicOpinion #SurveyResearch

w/ Berk @carohaensch.bsky.social @xinpeng.bsky.social Markus @barbaraplank.bsky.social Frauke @assenmacher.bsky.social

doi.org/10.48550/arX... preprint also out now on arxiv since srm has a bit of a backlog

We are waiting to get the proofs but I will send you the manuscript via Email in the meantimd

Leah von der Heyde Bernd Weiss and myself have an accepted manuscript at SRM doing this comparison for several LLMs plus a finetuned model

Johanna Hölzl from Uni Mannheim is working a lot on the reliability of Google Trends; she would for sure answer questions via Email

Similar to other preprints released in the last few months, we want to highlight the challenges of using LLMs for synthetic data creation or even public opinion estomation. While LLMs like GPT-3 hold promise for data synthesis, researchers must currently be very cautious of biases in predictions!

GPT-3's predictions often didn't match actual survey data from the GLES, with a notable variance in accuracy across different political parties. It performed better in predicting voters of center- and left-leaning parties than right-leaning ones. #syntheticdata

Our approach: We created detailed personas reflecting the German voting population, including demographic and political-ideological characteristics. These personas were used to prompt GPT-3 for voting behavior predictions. This approach is similar to a paper by Argyle et al. with ANES data. #LLM

Two researchers discussing "We can now use LLMs to impute oder synthesize values from surveys and other data collection types!" "This is a bad idea, LLMs were never built for such tasks" The author of the tweet says "Let's find out!"

First new research thread on Bluesky!

Leah von der Heyde's, Alexander Wenz's and my newest preprint examines the use of Large Language Models like GPT-3 in synthesizing public opinion data, focusing on German federal elections.

osf.io/preprints/so...

Challenges

More specific points from the feminist perspective

Important questions to ask

But as we say in germany, "da ist noch Luft nach oben" - we can do better

Some framework proposals already exist!

How can bias be mitigated when building systems?

Robots behaviour can serve as a validation for biased human behaviour!

Study asking: What is the average skin color for different professions in generative AI?

Will it all get worse with Generative AI? Probably yes

She first presents some examples from web search engines and other "old" systems

Hanan Salam at the University of Augsburg (HCAI workshop) on biases in AI Systems

An Alps landscape with lake water in the lower half of the picture, afternoon sun shining from behind the mountains, blue fall sky, dreamy atmosphere and a small boat before the shore

Conference posts coming soon but for now just another blue sky over the Königssee in Bavaria from this Sunday

The gothic town hall of Munich with a very blue and cloudless evening sky

Blue sky in Munich :)

Not surprised that academics chose to embrace the social media closest to a paywalled journal.

Artificial Intelligence, Unbiased Opinions? Assessing GPT’s suitability for estimating public opin... Poster presented at the 25th General Online Research Conference, September 20, Kassel, Germany. The recent development and release of widely accessible large language models (LLMs) appear to have th...

Due to the high interest in our GOR poster, we put the poster up on OSF :) von der Heyde, L., Wenz, A., & Haensch, A.-C. (2023). Artificial Intelligence, Unbiased Opinions? Assessing GPT’s suitability for estimating public opinion in multi-party systems doi.org/10.17605/OSF...

BlueSky Invites Please fill out this form if (1) you’d like to donate invite codes to scientists, clinicians, academics, and members of related communities or (2) you’re a member of one of these communities and w...

If you have invite codes for this place to spare, please think about donating them via the following service: docs.google.com/forms/d/e/1F...

Extremely useful:

A poster at a conference with the headline "GPT can't predict how Germans vote"

Sending my first skeet from #dgof_gor #gor23 :) Presented work with Leah von der Heyde and Alex Wenz on data collection through GPT-3! Feel free to take the poster home here: drive.google.com/file/d/1nrS_...