Matteo Tiezzi

Back to recurrent processing at the crossroad of transformers and state-space models - Nature Machine Intelligence While transformers and large language models excel at efficiently processing long sequences, new approaches have been proposed that incorporate recurrence to overcome the quadratic cost of self-attent...

🚨 Survey Alert 🚨

Our paper "Back to recurrent processing at the crossroad of transformers and state-space models" is out in Nature Machine Intelligence!

🔗 www.nature.com/articles/s42...
Read it here: rdcu.be/el85K

#Mamba #SSMs #Transformers #LLMs

(Top) A standard MLP catastrophically forgets samples learned online when data distribution shifts

(Bottom) A Memory Head is capable to avoid forgetting thanks to its learnable key-value routing mechanism. Keys become representative of the data distribution.

proceedings.mlr.press/v274/tiezzi2...

TL;DR: in Memory Heads, neurons route their computation across a learnable key-value mechanism
⚡️ dynamic behaviour depending on their own input.
⚡ Only the units(weights) that are relevant to process the observed sample are blended ➡ parameter isolation

proceedings.mlr.press/v274/tiezzi2...

@collasconf.bsky.social The #CoLLAs 2024 proceedings of our paper "Memory Head for Pre-Trained Backbones in Continual Learning" are out:

proceedings.mlr.press/v274/tiezzi2...

Video: lifelong-ml.cc/Conferences/...

🖥 Memory Head Code repository: github.com/sailab-code/...

A thread🧵

🌟 Best of Both Worlds!

❓Have you ever wondered why hybrid models (RNN + Transformers) are powerful? We answer this through the lens of circuit complexity and graphs!

Excited to share our work on understanding Graph Sequence Models (GSM), which allows the use of any sequence model for graphs.

LOT Spring School Topics

🐱 Learning Over Time (LOT) Spring School 📅 24-27 March 2025 | 📍 Siena, Italy
🔗 sites.google.com/unisi.it/lot...

Organizers:
Stefano Melacci, V. Lomonaco, Andrea Cossu, Alessandro Betti, @mtiezzi.bsky.social

#ContinualLearning #CL #LifelongLearning #CollectionlessAI

LOT Spring School - Registration PRE-REGISTRATION IS REQUIRED! Fill the form at the bottom of the page to pre-register now! Payment procedures will be activated in january 2025.

😸 Join the LOT Spring School with an incredible speaker lineup:
Battista Biggio, Nicolò Cesa-Bianchi, Elisa Ricci, Antonio Orvieto, Marco Gori, Tinne, Tinne Tuytelaars

🔗 sites.google.com/unisi.it/lot...

🔗 Pre-registrations are now open, secure your spot today: sites.google.com/unisi.it/lot...

LOT Spring School Topics

🐱 Learning Over Time (LOT) Spring School
📅 24-27 March 2025 | 📍 Siena, Italy

💡 Are you considering models that continuously adapt over time instead of learning "offline" from pre-designed-huge collections of data? 🚀

🔗 sites.google.com/unisi.it/lot...

#ContinualLearning #CL #LifelongLearning

Would love to join, working in CL and I was an organizer of CoLLAs 2024!

🌟 GLOW is coming back in December with amazing speakers: Emily Jin and @joshsouthern.bsky.social !

🗓️ Dec 18th @ 17 CET on Zoom, don't miss that!

🌐 Find more here: sites.google.com/view/graph-l...

Join this year's last edition of Graph Learning on Wednesdays (GLOW) tomorrow at 5pm CET!

Machine Learning on Graphs Join the conversation

Here is an initial starter pack list on Machine Learning on Graphs: go.bsky.app/HN2MTzp

May I join? Thanks

Entering the field, could you add me? Thanks!

✋

Starter pack for people in Lifelong and Continual Learning ✍️

@logconference.bsky.social #LOG meetup with @federico-errica.bsky.social open talk on "What is going on with oversmoothing, oversquashing, and underreaching?"

Wonderful location for the Siena #LoG meetup's poster session🤩

#LOG2024 Franco Scarselli kicking off the Learning on Graphs (LOG) Italian Meetup!!!

Working in Lifelong/Continual Learning, and I was among the organisers of last CoLLAs conference! Please add me☺️