Kimon Fountoulakis

Done.

I wrote a blog post about it.

Link: medium.com/@kimon.fount...

2) Can a neural network discover instructions for performing multiplication itself?

The answer to the first question is yes, with high probability and up to some arbitrary, predetermined precision (see the quoted post).

Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks Neural networks are known for their ability to approximate smooth functions, yet they fail to generalize perfectly to unseen inputs when trained on discrete operations. Such operations lie at the hear...

Link to the paper: arxiv.org/abs/2502.16763
Link to the repository: github.com/opallab/bina...

Learning to execute arithmetic exactly, with high probability, can be quite expensive. In the plot, 'ensemble complexity' refers to the number of independently trained models required to achieve exact learning with high probability. ell is the number of bits per number in the input.

New paper: Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks

Learning to execute arithmetic exactly, with high probability, can be quite expensive. In the plot, 'ensemble complexity' refers to the number of independently trained models required to achieve exact learning with high probability. ell is the number of bits per number in the input.

I never understood the point of trams. They're slow and expensive. I've been to two cities that built them while I was there, Edinburgh and Athens, and in both cases, the projects were born out of corruption. Especially in Edinburgh, it was a disaster. en.wikipedia.org/wiki/Edinbur...

Thanks, the connection to formal languages is quite interesting. I have a section in the repo regarding formal languages but it's small mainly because it's not a topic that I am familiar with. I will add them!

Update, 14 empirical papers added!

The SIAM Conference on Optimization 2026 will be in Edinburgh! I don’t really work on optimization anymore (at least not directly), but it’s cool to see a major optimization conference taking place where I did my PhD.

Currently NeurIPS has 21390 submissions. The final number last year was 15671.

Observation made by my student George Giapitzakis.

Got a pin this morning

Einstein problem: en.wikipedia.org/wiki/Einstei...

Our new work on scaling laws that includes compute, model size, and number of samples. The analysis involves an extremely fine-grained analysis of online sgd built up over the last 8 years of understanding sgd on simple toy models (tensors, single index models, multi index model)

Hey, I definitely predicted this correctly.

ChatGPT gives me the ability to expand my search capabilities on topics that I can only roughly describe, or even illustrate with a figure, when I don’t know the exact keywords to use in a Google search.

Positional Attention: Expressivity and Learnability of Algorithmic Computation There is a growing interest in the ability of neural networks to execute algorithmic tasks (e.g., arithmetic, summary statistics, and sorting). The goal of this work is to better understand the role o...

That's a comprehensive study on the expressivity for parallel algorithms, their in- and out-of-distribution learnability, and it includes a lot of experiments.

link: arxiv.org/abs/2410.01686

Positional Attention is accepted at ICML 2025! Thanks to all co-authors for the hard work (64 pages). If you’d like to read the paper, check the quoted post.

NeurIPS 2026 in the Cyclades. Just saying.

Wait, isn't that America?

A generalized neural tangent kernel for surrogate gradient learning State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful...

This is different from simply replacing the discontinuous activation in the neural network with a continuous one and then using standard NTK.

link: openreview.net/forum?id=kfd...

A generalized neural tangent kernel for surrogate gradient learning State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful...

They analyze a modified gradient flow, where the Jacobian for the training data uses an approximate derivative of the activation function.

I enjoyed reading the paper "A Generalized Neural Tangent Kernel for Surrogate Gradient Learning" (Spotlight, NeurIPS 2024).

They extend the NTK framework to activation functions that have finitely many jumps.

New AI model turns sketches into code | Cheriton School of Computer Science | University of Waterloo Co-developed by alum Ryan Yen, Code Shaping can transform coding beyond the keyboard.

✍️ Code Shaping, an AI-powered software, allows users to edit their code through sketches like diagrams and graphs 📈

🏆 This game-changing platform won the Best Paper Award at #CHI2025.

🔗Read more: uwaterloo.ca/computer-sci...

#UWaterloo #AI

Regarding this particular case, I can read exactly what the code is doing and it seems quite interpretable. I don't have to know some other library to understand the code. It seems to only use basic instructions.

I prefer more verbose coding, but again, I am only doing prototyping. I find it very annoying when someone's code includes efficient shortcuts that are hard to interpret without a lot of experience.

To be honest, I find the code by the chat way more readable, exactly because it's verbose. But, I suck at coding, I only prototype ideas...

Deterministic complexity analysis of Hermitian eigenproblems In this work we revisit the arithmetic and bit complexity of Hermitian eigenproblems. We first provide an analysis for the divide-and-conquer tridiagonal eigensolver of Gu and Eisenstat [GE95] in the ...

With my first Bluesky post, I am very pleased to share that my last PhD paper "Deterministic complexity analysis of Hermitian eigenproblems" has been accepted in ICALP 2025. A preprint is available on Arxiv:
arxiv.org/abs/2410.21550
A bit more info on linkedin: www.linkedin.com/posts/aleksa...