Our paper on the best way to add error bars to LLM evals is on arXiv! TL;DR: Avoid the Central Limit Theorem -- there are better, simple Bayesian and frequentist methods you should be using instead.
We also provide a super lightweight library: github.com/sambowyer/baye… 🧵👇
Go read it on arXiv! Thanks to my co-authors @sambowyer.bsky.social and @laurenceai.bsky.social 💥
📣 Jobs alert
We’re hiring postdoc and research engineer to work on UQ for LLMs!! Details ⬇️
#ai #llm #uq
Do you know what rating you’ll give after reading the intro? Are your confidence scores 4 or higher? Do you not respond in rebuttal phases? Are you worried how it will look if your rating is the only 8 among 3’s? This thread is for you.
Would love to be added!
But you can't prove that the *real* asteroid won't hit earth, because the real world isn't your simplified model. e.g. you don't know the initial conditions, there might be other bodies you aren't aware of etc. etc.
The analogy we're working from is "mathematically provable asteroid safety": within a simplified mathematical model, with known initial conditions, you can prove that an asteroid won't hit earth. (2/3)
Does anyone want to collaborate on an ICML position paper on "The impossibility of mathematically proving AI safety"? The basic thesis being that it is a category error to try to prove AI safety in the real world. (1/3)
Can you add?
