Adam Santoro

Research Engineer, Generative AI London, UK

My team is hiring! PRISM🌈 (Planning, Reasoning, Inference & Structured Models), which I co-lead with @theophane.bsky.social, is hiring a Research Engineer to help us develop planning and reasoning capabilities in Gemini. Apply here: boards.greenhouse.io/deepmind/job...

It's your unique toolset that will make your music authentic. So follow your own path, and be driven by achievements that you personally find valuable, whether that includes technical mastery on piano or not

Stripe Press — The Art of Doing Science and Engineering A groundbreaking treatise by one of the great mathematicians of our time, who argues that highly effective thinking can be learned.

You've probably read Hamming's essay You and Your Research, but perhaps not his book, The Art of Doing Science and Engineering. The edition by Stripe Press is absolutely stunning:

press.stripe.com/the-art-of-d...

A more charitable concern that I think fans have is that St Louis is spinning a modified version of the hybrid system, and is failing to communicate the changes he expects

Ya basically. It's then a question of how much data it would need to infer the meaning (via context or through parameter updates). If we think about how we'd speak and what we'd speak about in 1000 years then a current LLM would struggle mightily, even if the symbols are the same

Why does learning require changes to parameters? ICL in this regard is no different from meta-learning, wherein it's activation dynamics that underpin learning

I imagine this insight has a natural home in the embodied cognition school of thought. The same could probably be said about "attention", and arguably even "reasoning"

I'd argue the opposite: by virtue of being symbolic (i.e., meaning is established by convention, not by any fundamental and objective truth), it's trivial to go OOD with language

By virtue of being symbolic, language can always present new concepts or abstractions that aren't captured by any previous descriptions, making it trivial to conjure new OOD stuff

Re: the scale is dead debate. Isn't it pretty obvious that just scaling is never going to work if your method breaks down on OOD inputs? The world is non-stationary, so it's constantly presenting new OOD inputs.