Anirbit

@aifunmcr.bsky.social

Some luck to be hosted by a Godel Prize winner, Prof. Sebastien Pokutta, and to present our work in their group 💥 Sebastien heads this "Zuse Institute Berlin (#ZIB) " which is an amazing oasis of applied mathematics bringing together experts from different institutes in Berlin.

Interested in statistics? Prof Subhashis Ghoshal will be delivering the below public lecture tomorrow:

Title: Immersion posterior: Meeting Frequentist Goals under Structural Restrictions
Time: Aug 5 16:00-17:00
Abstract: www.newton.ac.uk/seminar/45562/
Livestream: www.newton.ac.uk/news/watch-l...

Hello #FAU. Thanks for the quick plan to host me and letting me present our exciting mathematics of ML in infinite-dimensions, #operatorlearning. #sciML Their "Pattern Recognition Laboratory" is completing 50 years! @andreasmaier.bsky.social 💥

@aifunmcr.bsky.social

University of Manchester has a 1 year post-doc position that I am happy to support in our group if you are currently an #EPSRC funded PhD student - and have the required specialization for work in our group. Typicall we prefer candidates who have published in deep-learning theory or fluid theory.

@aifunmcr.bsky.social

#aiforscience

DRSciML

Do mark your calendars for "DRSciML" (Dr. Scientific ML 😉) on September 9 and 10 🔥
drsciml.github.io/drsciml/
- We are hosting a 2 day international workshop on understanding scientific-ML.
- We have leading experts from around the world giving talks.
- There might be ticketing. Watch this space!

@aifunmcr.bsky.social

Major ML journals that have come up in the recent years,

- dl.acm.org/journal/topml
- jds.acm.org
- link.springer.com/journal/44439
- academic.oup.com/rssdat
- jmlr.org/tmlr/
- data.mlr.press

No reason why these cant replace everything the current conferences are doing and most likely better.

Thanks. No, AutoSGD is not going as far as delta-GClip goes. It's Theorem 4.5 is where they have any global minima convergence happening - but it uses assumptions which are not known to be true for nets. Our convergence holds for *all* nets wide enough.

Do link to the paper! I can have a look and check.

So, the next time you train a deep-learning model, it's probably worthwhile to have a baseline for the only provable adaptive gradient deep-learning algorithm - our delta-GClip 🙂

Our insight is to introduce an intermediate form of gradient clipping that can leverage the PL* inequality of wide nets - something not known for standard clipping. Given our algorithm works for transformers maybe that points to some yet unkown algebraic property of them. #TMLR

Our "delta-GCLip" is the *only* known adaptive gradient algorithm that provably trains deep-nets AND is practically competitive. That's the message of our recently accepted #TMLR paper - and my 4th TMLR journal 🙂

openreview.net/pdf?id=ABT1X...

#optimization #deeplearningtheory

GitHub - Anirbit-AI/Slides-from-Team-Anirbit: Slide Presentations of Our Works Slide Presentations of Our Works. Contribute to Anirbit-AI/Slides-from-Team-Anirbit development by creating an account on GitHub.

An updated version of our slides on necessary conditions for #SciML,
- and more specially,
"Machine Learning in Function Spaces/Infinite Dimensions".

Its all about the 2 key inequalities on slides 27 and 33.
Both come via similar proofs.

github.com/Anirbit-AI/S...

Now our research group has a logo to succinctly convey what we do - prove theorems about using ML to solve PDEs, leaning towards operator learning. Thanks to #ChatGPT4o for converting my sketches into a digital image 🔥 #AI4Science #SciML

It would be great to be able to see a compiles list of useful PDEs that #PINNs struggle to solve - and how would we measure success there.

We know of edge-cases with simple PDEs, where PINNs struggle, but then often those aren't the cutting-edge of use-cases of PDEs.

@prochetasen.bsky.social @mingfei.bsky.social @omarrivasplata.bsky.social

@aifunmcr.bsky.social 🙂

Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks We present and analyze a novel regularized form of the gradient clipping algorithm, proving that it converges to global minima of the loss surface of deep neural networks under the squared loss, provi...

A revised version of our delta-GClip algorithm - which is probably the *only* deep-learning algorithm that provably trains deep-nets while using step-size scheduling - and competes/supersedes heuristics like Adam and even Adam+Clipping on transformers.

arxiv.org/abs/2404.08624

He was the first person to interview me for a PhD position in applied maths and stats when I decided to shift my career focus in that direction. Years later when I became a faculty, he accepted my invite to fly to UK from Leipzig to give a talk and meet my students. Sayan is irreplaceable 😶

Fun @ @aifunmcr.bsky.social 😁

@aifunmcr.bsky.social 🙂

"data uncertainty" and "algorithmic randomness" are clearly 2 separate sources of uncertainty in ML predictions. So it makes sense to identify the former with EU and the later with AU?

My PhD student @dkumar9.bsky.social has done an amazing job of blending multiple deep results to establish that Langevin Monte-Carlo algorithm provably learns 2-layer nets for any data and for any size. This is very rigorously a "beyond NTK" regime. More to come from Dibyakanti 🙂

My first year PhD student Sébastien André-sloan presents at a #INFORMS conference in Toronto. Its a joint work with Matthew Colbrook at DAMTP, Cambridge. We prove a first-of-its-kind size requirement on neural nets for solving PDEs in the super-resolution setup - the natural setup for #PINNs.

The big surprise is that this isoperimetry kicks in via a "constant" amount of regularization - something that doesn't scale with size of the nets. (2/2)

Our key idea is to figure out that neural losses at arbitrary size and data can induce Gibbs' measures which satisfy the Poincare inequality. After that an existing result of Michael Jordan-Weijie Su-Bin Shi takes over (or any of the LMC results). (1/2)