Kawin Ethayarajh

Interesting. Is this because they have govt contracts and those would be jeopardized? Or uncertainty around whether those models will be banned for use in America, which adds a huge risk premium?

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences. Although these methods are designed to teach a model to generate prefer...

relevant paper: arxiv.org/abs/2410.08847

for all methods, it is better for data to be on-policy and be labelled as good/bad relative to the current state of the policy.

but ultimately this is a learning dynamics problem that transcends how the data is sampled

unpaired methods work the way we hope paired methods would, simultaneously increasing the relative prob of good outputs and decreasing the relative prob of bad outputs. this allows you to skip SFT entirely

it's not really on vs. off-policy. in theory, paired methods should increase the prob of good outputs, decrease prob of bad outputs. in practice, they decrease *both*. you need to do SFT beforehand so that you can pay this price and hope that relative to the base model, p(good|x) is still higher

all paired preference methods suffer from this problem while also being more inflexible. unpaired preference methods are always the way to go IME

Source: old.reddit.com/r/LocalLLaMA...

🤔

Evolution of Preference Optimization Techniques | Hippocampus's Garden RLHF is not the only method for AI alignment. This article introduces modern algorithms like DPO and KTO that offer simpler and more stable alternatives.

RLHF is not the only method for AI alignment. This post introduces modern algorithms like DPO, KTO, and DiscoPOP that offer simpler and more stable alternatives.

Evolution of Preference Optimization Techniques | Hippocampus's Garden
hippocampus-garden.com/preference_o...

Reference-Based Metrics Are Biased Against Blind and Low-Vision Users’ Image Description Preferences Rhea Kapur, Elisa Kreiss. Proceedings of the Third Workshop on NLP for Positive Impact. 2024.

I'm excited to kick off my Bluesky presence with wonderful news: Our paper "Reference-Based Metrics Are Biased Against Blind and Low-Vision Users' Image Description Preferences" won a Best Paper Award at the NLP for Positive Impact Workshop at EMNLP! Read it here: aclanthology.org/2024.nlp4pi-...

nominating myself @kawinethayarajh.bsky.social

These differences with the DPO version don't seem statistically significant?

(almost) all good poetry has high perplexity. it's by design something an out-of-the-box llm would be bad at. alignment on one poet would actually help imo.

Moderately grumpy UToronto alumnus nominating myself 🙋

Could i be added (recent alumnus)? Thank you!

Everyone is fixated on replicating o1 when there would be way more utility in figuring out what makes Claude so special.

Would love to be added thanks!