Andrew Gordon Wilson

YouTube video by The Piano Experience Glenn Gould plays Chopin Piano Sonata No. 3 in B minor Op.58

Gould mostly recorded baroque and early classical. He only recorded a single Chopin piece, as a one-off broadcast. But like many of his efforts, it's profoundly thought provoking, the end product as much Gould as it is Chopin. I love the last mvt (20:55+). www.youtube.com/watch?v=NAHE...

I don't think those things seem boring. But most research directions honestly are quite boring, because they are geared towards people pleasing --- going with the herd, seeking approval from others, and taking no risks. It's a great way to avoid making a contribution that changes any minds.

Whatever you do, just don't be boring.

YouTube video by LoG Meetup NYC It's Time to Say Goodbye to Hard (equivariance) Constraints - Andrew Gordon Wilson

I had a great time presenting "It's Time to Say Goodbye to Hard Constraints" at the Flatiron Institute. In this talk, I describe a philosophy for model construction in machine learning. Video now online! www.youtube.com/watch?v=LxuN...

Excited to be presenting my paper "Deep Learning is Not So Mysterious or Different" tomorrow at ICML, 11 am - 1:30 pm, East Exhibition Hall A-B, E-500. I made a little video overview as part of the ICML process (viewable from Chrome): recorder-v3.slideslive.com#/share?share...

While scaling laws typically predict the final loss, we show that good scaling rules enable accurate predictions of entire loss curves of larger models from smaller ones! Shikai Qiu did an amazing job leading the paper, in collaboration with L. Xiao, J. Pennington, A. Agarwala. 3/3

In particular, scaling collapse allows us to transfer insights from experiments conducted at a very small scale to much larger models! Much more in the paper, including supercollapse: collapse between curves less than the noise floor of per-model loss curves across seeds. 2/3

Our new ICML paper discovers scaling collapse: through a simple affine transformation, whole training loss curves across model sizes with optimally scaled hypers collapse to a single universal curve! We explain the collapse, providing a diagnostic for model scaling.
arxiv.org/abs/2507.02119
1/3

I love havarti!

It sometimes feels like papers are written this way. <Make claim that may or may not be true but aligns with the paper's narrative> <find arbitrary reference that supposedly supports that claim, but may be making a different point entirely>. I guess grammarly is giving the people what they want?

Excited about our new ICML paper, showing how algebraic structure can be exploited for massive computational gains in population genetics.

Machine learning is perhaps the only discipline that has become less mature over time. A reverse metamorphosis, from butterfly to caterpillar.

AI this, AI that, the implications of AI for X... can we just never talk about AI again?

Really excited about our new paper, "Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion". We explain the mysterious success of masking diffusion to propose new diffusion models that work well in a variety settings, including proteins, images, and text!

What's irrational is the idea that some group of authors writing a paper about something foundational also should be the team of people to put it into production in the real world and demonstrate its impact, all in one paper. That happens over years, and involves different interests and skills.

I find that this position is often more emotionally rooted than rational. It makes no sense to expect a paper on foundations to demonstrate significant real-world impact. As you say, it's a cumulative process carried out by different people, over time.

Sorry to miss it! Currently in Cambridge UK for a Newton Institute programme on uncertainty representation.

Great topic for a workshop!

YouTube video by Lex Fridman Terence Tao: Hardest Problems in Mathematics, Physics & the Future of AI | Lex Fridman Podcast #472

A really outstanding interview of Terence Tao, providing an introduction to many topics, including the math of general relativity (youtube.com/watch?v=HUkB...). I love relativity, and in a recent(ish) paper we also consider the wave maps equation (section 5, arxiv.org/abs/2304.14994).

AI benchmarking culture is completely out of control. Tables with dozens of methods, datasets, and bold numbers, trying to answer a question that perhaps no one should be asking anymore.

We have a strong bias to overestimate the speed of technological innovation and impact. See past claims about autonomous driving, AI curing diseases... or the timeline in every sci-fi book ever written. Where is my flying car?

Mad not to have Hey Jude or Yesterday.

Still not sure I totally understand what you're saying. Why is it unfortunate that it doesn't fit with academia or industry?

That said, if you're saying that there are a space of projects that have value but that value is not easily recognized with traditional structures (academia, industry), I certainly agree.

It all depends. GPyTorch, BoTorch, and CoLA have not only facilitated but actually suggested to us some of our best research ideas. Google and Meta also must feel they benefit from Tensorflow, JAX, and PyTorch being open source.

Can you give some examples of these sorts of systems?

Did you invoke "number of atoms in the universe"? :)

The textbooks don't need to be rewritten -- they just needed to pay attention to what was already known about generalization, decades ago! I've had thoughts about this for 12 years, and people always ask for the paper -- so I finally wrote it. Thankful to many for feedback! 12/12

We also further consider overparametrization and double descent. Deep learning of course _is_ different, and much is not well understood. To this end, we particularly highlight representation learning, mode connectivity, and universal learning. Much more in the paper! 11/12

Computing the bounds is also very simple: (i) train a model to find hypothesis h*, using any optimizer; (ii) measure the empirical risk R(h*) (e.g., training loss); (iii) measure the filesize of the stored model. 10/12