When using LLM-as-a-judge, practitioners often use greedy decoding to get the most likely judgment. But we found that deriving a score from the judgment distribution (like taking the mean) works better!
❌LLM-as-a-judge with greedy decoding
😎Using the distribution of the judge’s labels
06.03.2025 22:04 — 👍 28 🔁 4 💬 1 📌 0
Announcement #1: our call for papers is up! 🎉
colmweb.org/cfp.html
And excited to announce the COLM 2025 program chairs @yoavartzi.com @eunsol.bsky.social @ranjaykrishna.bsky.social and @adtraghunathan.bsky.social
17.12.2024 15:48 — 👍 67 🔁 24 💬 0 📌 1
I'm at #Neurips2024 this week!
My work (arxiv.org/abs/2406.17692) w/ @gregdnlp.bsky.social & @eunsol.bsky.social exploring the connection between LLM alignment and response pluralism will be at pluralistic-alignment.github.io Saturday. Drop by to learn more!
11.12.2024 17:39 — 👍 28 🔁 6 💬 0 📌 0
The 2025 Conference on Language Modeling will take place at the Palais des Congrès in Montreal, Canada from October 7-10, 2025
Research Scientist @ Mosaic x Databricks. Adaptive Methods for Retrieval, Generation, NLP, AI, LLMs https://mrdrozdov.github.io/
PhD Student at Johns Hopkins University. Previously: Allen Institute for AI, Apple, Samaya AI. Research for #NLProc #IR
Associate Professor in Computer Science at the University of Maryland. Human-Centered Natural Language Processing & Machine Translation
Human-centered AI #HCAI, NLP & ML. Director TRAILS (Trustworthy AI in Law & Society) and AIM (AI Interdisciplinary Institute at Maryland). Formerly Microsoft Research NYC. Fun: 🧗🧑🍳🧘⛷️🏕️. he/him.
Research Scientist, Google DeepMind / Ex-academic / Deep learning to help people write code / ❤️s:🐱🐶☕️🍕
#nlp researcher interested in evaluation including: multilingual models, long-form input/output, processing/generation of creative texts
previous: postdoc @ umass_nlp
phd from utokyo
https://marzenakrp.github.io/
Postdoc in AI at the Allen Institute for AI & the University of Washington.
🌐 https://valentinapy.github.io
Assoc. Prof in CS @ Northeastern, NLP/ML & health & etc. He/him.
Incoming Asst Prof @UMD Info College, currently postdoc @UChicago. NLP, computational social science, political communication, linguistics. Past: Info PhD @UMich, CS + Lx @Stanford. Interests: cats, Yiddish, talking to my cats in Yiddish.
PhD @ MIT. Prev: Google Deepmind, Apple, Stanford. 🇨🇦 Interests: AI/ML/NLP, Data-centric AI, transparency & societal impact
Asst prof @ University of Utah · NLP · she/her 🇭🇷
natural language processing and computational linguistics at google deepmind.
Assistant professor at https://si.umich.edu/ working in computational social science, machine learning, and NLP | https://dallascard.github.io
A LLN - large language Nathan - (RL, RLHF, society, robotics), athlete, yogi, chef
Writes http://interconnects.ai
At Ai2 via HuggingFace, Berkeley, and normal places
NLP/AI Research
Assistant Professor @Yale
AI @ OpenAI, Tesla, Stanford
jmhessel.com
NLP PhD; Seattle bike lane enjoyer; posts about machine learning, language processing, computer vision, transit
AI, RL, NLP, Games Asst Prof at UCSD
Research Scientist at Nvidia
Lab: http://pearls.ucsd.edu
Personal: prithvirajva.com
