Elinor🎗️ @ COLM 🍁

Evaluation reasoning interpretability rl in context benchmark alignment synthetic data

COLM word cloud. Yoav says it’s the year of reasoning, but evaluation is also huge.

bsky.app/profile/elin...

I’m at #COLM2025! Would love to chat about anything related to pluralistic alignment, fairness evaluations, societal impacts of LLMs, etc 😊

You can also find me at the NLP4Democracy workshop giving a talk about my work analyzing democratic deliberation with LLMs Oct 10th!

Alright the evening sky, you’re utterly wondrous and fantastical, we get it, geez

Here’s a #COLM2025 feed!

Pin it 📌 to follow along with the conference this week!

NLP 4 Democracy - COLM 2025

I will be at #COLM2025 this week, and would love to connect with folks interested in applications (and critiques) of language modeling in social science research!

And join us for the NLP4Democracy workshop on Friday!

sites.google.com/andrew.cmu.e...

#NLP #NLProc #LLM #ComputationalSocialScience

Very excited that my paper with @katakeith.bsky.social is now out in @polanalysis.bsky.social. We investigate whether LLMs actually follow the instructions/definitions provided in codebooks, propose some diagnostics, and release a new evaluation dataset.
www.cambridge.org/core/journal...

I wish students understood in most empirical AI research there’s a huge scientific advantage from being constitutionally excited by math vs intimidated, but very little additional gain from being actually “good” at math. Maybe they’d be less intimidated if they didn’t feel they had to be “good”.

An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies In an era of increasing societal fragmentation, political polarization, and erosion of public trust in institutions, representative deliberative assemblies are emerging as a promising democratic forum...

This work is part of my master’s thesis at @mit.edu @medialab.bsky.social, supervised by Deb Roy and with the help of Jad Kabbara @jad-kabbara.bsky.social.

🔗 arxiv.org/abs/2509.12577

Beyond research, this paves the way for:
✨ Tools supporting live assemblies in real time
✨ Increasing transparency & communicating critical insights to decision-makers
✨ Enabling richer cross-assembly analysis to advance research on deliberative best practices

In the tech-enhanced assembly, our framework revealed:
🔹 How deliberation surfaced, refined, or discarded ideas
🔹 *Missing* viable ideas
🔹 How opinion shifts & rec edits shaped outcomes
🔹 Underlying values & trade-offs invisible to decision-makers

We develop an LLM-based framework to:
✅ Map how suggestions transform into concrete recommendations
✅ Reconstruct individuals’ evolving perspectives
✅ Detect why votes shift across deliberation

Despite their promise, we still lack tools to empirically trace:
• how ideas evolve into recommendations
• how deliberation shapes perspectives & votes

At MIT CCC, we hosted our own tech-enhanced assembly to explore how AI can help!

sustainabilityassembly.portal.cortico.ai

Deliberative assemblies bring together everyday citizens selected by lottery. Through deliberation 💬 & learning, they collectively form policy recommendations 💡for decision-makers.

They’ve proven successful worldwide, facilitating rebuilding trust & strengthening democracy 🤝.

An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies In an era of increasing societal fragmentation, political polarization, and erosion of public trust in institutions, representative deliberative assemblies are emerging as a promising democratic forum...

🚨 New preprint! 🚨
Excited to share my work: An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies 🤖🗳️

I’ll be presenting this at @colmweb.org in the NLP4Democracy workshop!

🔗 arxiv.org/abs/2509.12577

"This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity."

super interesting

We must build AI for people; not to be a person

This paper yields the same conclusion as what @mustafasuleymanai.bsky.social recently posted on the danger of 'seemingly conscious AI'.

mustafa-suleyman.ai/seemingly-co...

Thread inspired by having to review 6(!!) papers for AAAI and most of them having no line numbers. And one particularly great paper I want to show the authors exactly how much I enjoyed it via my annotation drawings (>20 check marks, ~10 exclamations, and even 2 hearts!)

Typing the (sometimes extreme) number of typo corrections is tedious, time consuming, and especially frustrating when there’s no line numbers on the pdf! It would honestly be faster for me to edit it myself 🙄

When reading papers, especially reviewing, I like to print and annotate as I read. I wish I could upload this to open review so authors can see smaller suggestions (typos, formatting errors) as well as smaller positive notes eg things I appreciated or found useful/interesting

!!!!! me too

i'm curious to hear your thoughts on their piloting of AI assisted reviewing: bsky.app/profile/aaai...

AAAI-26 Review Process Update: Scale, Integrity Measures, and Pathways to Sustainability - AAAI

totally agree! the peer review system is already over burdened and there needs to be an intermediate step for AI generated work. for example, AAAI received *double* the amount of normal submissions this year even after desk rejections aaai.org/conference/a....

First crochet project done! Super proud and excited for the next one

Center for the Alignment of AI Alignment Centers We align the aligners

This new center strikes the right tone in approaching the AI alignment problem. alignmentalignment.ai

Yeah absolutely. I wasn’t sure how much of the modelling work you were aiming for, but thought to share just in case :))
It would be interesting also to check if cultural alignment work fully “aligns” w the goals of cultural analytics researchers too

Benchmarking Vision Language Models for Cultural Understanding Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has bee...

maybe this one is relevant: "Benchmarking Vision Language Models for Cultural Understanding"

arxiv.org/abs/2407.10920

It never ceases to amaze me that some folks can just continue on with their evenings and not stop everything to gaze at the magnificence of the post-rain sunset (and ofc take a billion photos)!

This is a cool paper that suggests that AI agents can indeed be used for social science experiments, but that just using a chatbot isn't good enough, instead prompts developed based on social & game theory makes AI agent actions predictive of real human outcomes. benjaminmanning.io/files/optimi...

im literally doing this rn, with the submission deadline being tomorrow 😅