T. Anderson Keller

Flow Equivariant Recurrent Neural Networks - Kempner Institute Sequence transformations, like visual motion, dominate the world around us, but are poorly handled by current models. We introduce the first flow equivariant models that respect these motion symmetrie...

New in the #DeeperLearningBlog: #KempnerInstitute research fellow @andykeller.bsky.social introduces the first flow equivariant neural networks, which reflect motion symmetries, greatly enhancing generalization and sequence modeling.

bit.ly/451fQ48

#AI #NeuroAI

(1/7) New preprint from Rajan lab! 🧠🤖
@ryanpaulbadman1.bsky.social & Riley Simmons-Edler show–through cog sci, neuro & ethology–how an AI agent with fewer ‘neurons’ than an insect can forage, find safety & dodge predators in a virtual world. Here's what we built

Preprint: arxiv.org/pdf/2506.06981

What shapes the topography of high-level visual cortex?

Excited to share a new pre-print addressing this question with connectivity-constrained interactive topographic networks, titled "Retinotopic scaffolding of high-level vision", w/ Marlene Behrmann & David Plaut.

🧵 ↓ 1/n

Are you an RL PhD at Harvard who has had your funding wrecked by the government and working on topics related to multi-agent? Reach out, I am happy to try to find a way to support you.

Looking forward to presenting our work on cortico-hippocampal coupling and wave-wave interactions as a basis for some core human cognitions

5pm May 6th EST (US)
8am May 7th AEST (Sydney)

Zoom link: columbiacuimc.zoom.us/j/92736430185

Thanks to WaveClub conveners Erfan Zabeh & Uma Mohan

It’s another big day for the #KempnerInstitute at @CosyneMeeting! Check out our work highlighted in poster session 3 today! #COSYNE2025

Such a cool connection!! I never heard of that, but that is an ingenious solution. I will likely use this reference in my future talks and mention your comment if you don’t mind!

Thanks for reading! Can you explain your thought process here? Imagine a neuron with a receptive field (size of the yellow square) localized to the center of the pentagon. Its input would be entirely white — same as if it were localized to the center of the triangle; and therefore indistinguishable.

Super interesting thread!

And not to forget, a huge thanks to all those involved in the work: Lyle Muller, Roberto Budzinski & Demba Ba!! And further thanks to those who advised me and shaped my thoughts on these ideas @wellingmax.bsky.social & Terry Sejnowski. This work would not have been possible without their guidance.

Traveling waves of neural activity are observed all over the brain. Can they be used to augment neural networks?

I am thrilled to share our new work, "Traveling Waves Integrate Spatial Information Through Time" with @andykeller.bsky.social!

1/13

Really interesting RNN work.

And based on some spiking simulations I've tinkered with, it seems plausible that PV, CB & CR interneurons can contribute to changing the boundary conditions and the 'elasticity' of the oscillating 'rubber sheet' of cortex (and probably hippocampus and amygdala too). 🤓

Traveling Waves Integrate Spatial Information Through Time Traveling waves of neural activity are widely observed in the brain, but their precise computational function remains unclear. One prominent hypothesis is that they enable the transfer and integration...

For all the technical details and more ablations, please see our paper recently accepted in workshop-form at ICLR Re-Align, and full-version preprint on ArXiv!

Paper: arxiv.org/abs/2502.06034
Code: github.com/KempnerInsti...

Hope to see you in Singapore!

Fin/

Traveling Waves Integrate Spatial Information Through Time - Kempner Institute The act of vision is a coordinated activity involving millions of neurons in the visual cortex, which communicate over distances spanning up to centimeters on the cortical surface. How do […]

If you want more visualizations, a bit more depth, and even some audio of what different images 'sound' like to our models, please check out our @kempnerinstitute.bsky.social blog-post!

kempnerinstitute.harvard.edu/research/dee...

13/14

Overall, we believe this is the first step of many towards creating neural networks with alternative methods of information integration, beyond those that we have currently such as network depth, bottlenecks, or all-to-all connectivity, like in Transformer self-attention.

12/14

Tables from the paper comparing wave based models and baselines (CNNs and U-Nets) on a variety of semantic segmentation tasks

We found that wave-based models converged much more reliably than deep CNNs, and even outperformed U-Nets with similar numbers parameter when pushed to their limits. We hypothesize that this is due to the parallel processing ability that wave-dynamics confer and other CNNs lack.

11/14

As a first step towards the answer, we used the Tetris-like dataset and variants of MNIST to compare the semantic segmentation ability of these wave-based models (seen below) with two relevant baselines: Deep CNNs w/ large (full-image) receptive fields, and small U-Nets.

10/14

Horizontal Propagation of Visual Activity in the Synaptic Integration Field of Area 17 Neurons The receptive field of a visual neuron is classically defined as the region of space (or retina) where a visual stimulus evokes a change in its firing activity. At the cortical level, a challenging is...

We were super excited about these results—they aligned with the long-standing hypothesis that traveling waves integrate spatial information in the brain*. But does this hold any practical implications for modern machine learning?

pubmed.ncbi.nlm.nih.gov/7947408
www.science.org/doi/abs/10.1...

9/14

Was this just due to using Fourier transforms for semantic readouts, or wave-biased architectures? No! The same models with LSTM dynamics and a linear readout of the hidden-state timeseries still learned waves when trying to semantically segment images of Tetris-like blocks!

8/14

Plot of five representative frequency bins from the FFT of the dynamics of our wave-RNN on the shape task. We see different shapes pop out in different bins, indicating that they 'sound' different, and allowing the model to uniquely classify each shape. On the right we plot the average FFT for each pixel, separated by each shape, over the whole dataset, showing that different shapes do have measurably different frequency spectra, even in this average case.

Looking at the Fourier transform of the resulting neural oscillations at each point in the hidden state, we then saw that the model learned to produce different frequency spectra for each shape, meaning each neuron really was able to 'hear' which shape it was a part of!

7/14

We made wave dynamics flexible by adding learned damping and natural frequency encoders, allowing hidden state dynamics to adapt based on the input stimulus. On simple polygon images, we found the model learned to use these parameters to produce shape-specific wave dynamics:

6/14

Visualization of the input stimuli to our network (left) and the target segmentation labels by color (right). The receptive field of the final layer neurons in our model is plotted as the yellow box, demonstrating that a single neuron has no way to know what shape it may be a part of simply from its local neighborhood, and therefore will require global integration of information over time to solve the task.

To test this, we needed a task; so we opted for semantic segmentation on large images, but crucially with neurons having very small one-step receptive fields. Thus, if we were able to decode global shape information from each neuron, it must be coming from recurrent dynamics.

5/14

Visualization of the same wave-based RNN on two drums of different sizes (13 and 33 side length respectively). In the middle (in purple) we show the displacement of the drum head at a point just off the center, and (in red) the theoretical fundamental frequency of vibration that we can analytically derive for a square of side length L plotted. On the right we show the Fourier transform of these time-series dynamics, showing the frequency peak in the expected location. This validates we can estimate the size of a drum head from the frequency spectrum of vibration at any point.

We found that, in-line with theory, we could reliably predict the area of the drum analytically by looking at the fundamental frequency of oscillations of each neuron in our hidden state. But is this too simple? How much further can we take it if we add learnable parameters?

4/14

Inspired by Mark Kac’s famous question, "Can one hear the shape of a drum?" we thought: Maybe a neural network can use wave dynamics to integrate spatial information and effectively "hear" visual shapes... To test this, we tried feeding images of squares to a wave-based RNN:

3/14

Traveling Waves in Visual Cortex In this Review, Sato et al. summarize the evidence in favor of traveling waves in primary visual cortex. The authors suggest that their substrate may lie in long-range horizontal connections and that ...

Just as ripples in water carry information across a pond, traveling waves of activity in the brain have long been hypothesized to carry information from one region of cortex to another (Sato 2012)*; but how can a neural network actually leverage this information?

* www.cell.com/neuron/fullt...
2/14

In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network?

Super excited to share recent work with the brilliant @mozesjacobs.bsky.social: "Traveling Waves Integrate Spatial Information Through Time"

1/14

New research shows neurons learn to encode and transmit information to other spatially distant neurons through traveling waves. Read more in the #KempnerInstitute’s blog: bit.ly/3DrIPEq