Ahmad Beirami

The math that LLMs can do today is novel enough to be considered publishable, but it's not the kind of math that would be consequential.

My thoughts on the broken state of AI conference reviewing: Years ago, when I was in graduate school and a postdoc in Information Theory, I always felt fortunate to be invited to review for IEEE… | A... My thoughts on the broken state of AI conference reviewing: Years ago, when I was in graduate school and a postdoc in Information Theory, I always felt fortunate to be invited to review for IEEE Tran...

My thoughts on the broken state of AI conference reviewing.

www.linkedin.com/feed/update/...

Let's regress from here to AGI!

Slide titled “Takeaways (alignment recipe).” Step 1: Perform Best-of-n and make sure it works as desired. – Inspect a few responses and verify the reward-induced ranking makes sense. – Best-of-n gives the best trade-offs; if it doesn’t work, no fancy method will. – You can debug best-of-n much faster. Step 2: Only then train your favorite alignment method. – Track KL(π‖p) throughout training: • KL > 100: results are unlikely to be useful. • KL > 15: inspect outcomes for reward hacking. • KL < 8: you are probably OK. Bottom banner in a black box repeats “(1) Look at your data! (2) Look at your data! (3) Look at your data!” in blue, green, and red.

This is the conclusion slide of a talk I gave more than a year ago on RL/Alignment! It still holds true today.

This also applies to telling your story (e.g., in a CV, bio, interview, etc).

Focus on what you have accomplished and what you are excited about doing next; not just where you did it!

Haha. the content was:

If a paper is great, the credit goes to the first author.

If a paper has any flaws, the responsibility falls on the last author.

The actual unpopular opinion is that the notion of senior and junior authors should be abolished. It has completely diluted the notion of scientific authorship and created this entire industry of free-riding, head-in-the-clouds, incompetent PIs/managers. List down exact contributions instead. [+]

Glad you asked.
bsky.app/profile/abei...

I occasionally get messages asking how to follow my path and get into Meta, DeepMind, or similar places. That is the wrong question. Do not focus on the brand! Focus on what you want to work on, then find the opportunity that fits your goals best.

Related to this if a paper turns out to have a major error in it, you’re supposed to throw yourself under the bus not your students.

I think for every deliverable, there has to be one person who is responsible (gets it done) and one person who is accountable (makes sure it's done correctly).

Middle authors can be responsible or accountable for a subset of tasks.

Corollary: If you lack bandwidth or expertise to act as the verifier, then you shouldn&#39;t sign up to be the senior author of a paper! | Ahmad Beirami Corollary: If you lack bandwidth or expertise to act as the verifier, then you shouldn't sign up to be the senior author of a paper!

proposed contribution breakdown by Atlas Wang makes a lot of sense imo:
www.linkedin.com/feed/update/...

Ahmad Beirami on X: "Good time to remind ourselves that: If a paper is great, the credit goes to the first author. If a paper has any flaws, the responsibility falls on the last author." / X Good time to remind ourselves that: If a paper is great, the credit goes to the first author. If a paper has any flaws, the responsibility falls on the last author.

Not really. I've been saying variants of the same thing for a long time:

x.com/abeirami/sta...

Corollary: If you lack bandwidth or expertise to act as the verifier, then you shouldn't sign up to be the senior author of a paper!

The junior author is the generator. The senior author is the verifier. The verifier should teach/distill some checks to the generator, but the verifier keeps final responsibility. If a wrong claim gets out, it is on the verifier!

Unpopular opinion:
When a paper has a senior mentor and a junior mentee, the senior author must make sure the claims are correct and well supported. They must check every claim and gate the submission until it meets that bar.

This is the recipe for many provable claims:

Make enough assumptions and narrow down the claim, then prove a narrow result with caveats. Present it as broad, hide the caveats, and declare “XYZ is provable!”

Today, Industry research is focused on short term (3-6months) bets. Academics have an opportunity to balance their portfolio with medium term (1-2 years) and long term (5-10 years) bets. Putting all academic efforts in short-term basket is suboptimal!

When I worked in corporate, I was often first in the office because that routine worked for me. It was a personal preference, not a benchmark for anyone else.

We should not judge commitment by hours, especially in research. We should look for thoughtful work and steady progress.

True, this sounds obvious but it is more common than we'd hope, unfortunately.

Common mistake in LLM prompting projects: jumping into full-scale pipelines (datasets and inference) without testing feasibility. Iterating at scale is expensive and time-consuming.

Start with ONE example to validate the hypothesis, verify context, debug the design, then scale.

I have to admit that I embarrassingly didn't know about this history. Nice reading material for the long weekend, thank you :)

But I don't disagree that maybe it is already better than an average human, because we humans are pretty bad at this too.

And audience modeling is a hard problem and won't be solved unless we effectively solve memory.

On the clarity point: I think we are still not there yet on info seeking queries, e.g., GPT5 Thinking returns a lot of info (high recall) but is unable to put the info in a clear and logical structure.

The open question is how to choose annotators who can teach that skill. We need people who understand the point in the first place, and who can tell when a point is vague, what context is missing, and how to fix it.

Thoughts that are explained clearly are more respected. Clarity is a scarce skill. Many of us (me included) leave out key context and make people work too hard to understand us.

AI models should get better at this: not just reasoning, but communicating with the right amount of context.

YouTube video by Delta Institute Ep. 23: Ahmad Beirami, Google DeepMind Former Research Lead

Episode link: www.youtube.com/watch?v=ZMH7...

Enjoyed speaking with @DeltaInstitutes about going from information theory to ML, recent safety alignment/RL work, and lessons on RL for LLMs that stuck!

Check out the podcast episode here: lnkd.in/eb6dWHDv

In 2009, a prominent signal processing professor said the market was tough and h-index ≥6 was needed just to get a faculty interview.

We now seem to be drifting toward the same bar for PhD program entrance.

‎Gemini - Recreating Art in Identical Style Created with Gemini

See the conversation:
gemini.google.com/share/1ffefd...