Ben Walker

Really enjoyed chatting with @oxfordmathematics.bsky.social
about AI in mathematics, where it can genuinely help, and what some of the limitations are.

Alternative Title: ‘A Timely Series of Talks on Time Series.’

Was too proud of this one so had to post it somewhere!

Just wrapped up my short course ‘Time Series Modelling: From Foundations to Frontiers’ at the Oxford Internet Institute @oii.ox.ac.uk

Huge thanks to @ammaox.bsky.social for the invitation—had some really engaging discussions!

Looking forward to being back at the OII soon.

Sonnet-3.7 has me vibe coding for the first time 🎧💻

Never written html, css, or JavaScript before, but I’ve created the website I’ve always wanted, featuring an optional command line interface ✨

BenWalker.co.uk

Looking forward to presenting this work at #NeurIPS2024 !

Come find us on Thursday from 11-2 @ West Ballroom A-D #6907

D&D Combinatorics xkcd.com/3015

Huge thanks to my incredible co-authors Nicola Cirone, Antonio Orvieto, Cristopher Salvi, and Terry Lyons!

#NeurIPS2024 #MachineLearning #DeepLearning #StateSpaceModels

🧵6/6

S4, Mamba, and Transformers need 4 blocks just to compose 12 permutations!

In contrast, using a dense state-transition matrix (IDS4/Linear CDE) or a non-linear state-transition (RNN) allows for state-tracking with only 1 layer.

🧵5/6

An excellent empirical example of this limited capacity is the A5 benchmark, from “The Illusion of State in State-Space Models” by Merrill et al.

The benchmark tests state-tracking, a crucial ability for tasks involving permutation composition like chess.

The results? 👇

🧵4/6

We rigorously show that Mamba’s selectivity mechanism boosts expressiveness.

However, we also show that using a diagonal state-transition matrix—while drastically reducing computational costs—also significantly limits the model's capacity.

🧵3/6

In this paper, we introduce a unified framework for state-space models using Rough Path Theory, providing a rigorous theoretical foundation for why the Mamba recurrence outperforms other SSMs—and precisely where their expressiveness may be limited.

🧵2/6

Theoretical Foundations of Deep Selective State-Space Models Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstan...

Want to know why Mamba beats other state-space models—and where it falls short?

Then check out our #NeurIPS 2024 paper: "Theoretical Foundations of Deep Selective State-Space Models."

🔗 Read the paper: arxiv.org/abs/2402.19047
💻 Access the code: github.com/Benjamin-Walke…

🧵1/6