Riccardo Grazzi

I also think that the analysis might be further enhanced thanks to the very nice results in the RWKV-7 paper!

In our DeltaProduct work we also add a bit of theory to DeltaNet, showing that it can solve Dihedral groups, which are the groups of symmetries of regular polygons, with only two layers. This includes S3 (symmetries of the equilateral triangle).

I just had the pleasure of presenting "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues" at
the AutoML Seminars. I also presented some proof sketches :)

Here's the recording: t.co/B3TQozvGKX

Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers in large language modeling, offering linear scaling with…

Excited to present
"Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues"
at the M3L workshop at #NeurIPS
https://buff.ly/3BlcD4y

If interested, you can attend the presentation the 14th at 15:00, pass at the afternoon poster session, or DM me to discuss :)

At #NeurIPS2024 🇨🇦 our group will present 7 contributions! These span a diverse array of topics: from theoretical advances in stochastic processes and reinforcement learning to applications in molecular dynamics and uncertainty quantification.