MAMBA and State Space Models explained | SSM explained

We simply explain and illustrate Mamba, State Space Models (SSMs) and Selective SSMs.
SSMs match performance of transformers, but are faster and more memory-efficient than them. This is crucial for long sequences!

Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏
Dres. Trost GbR, Siltax, Vignesh Valliappan, Michael

Outline:
00:00 Mamba to replace Transformers!?
02:04 State Space Models (SSMs) – high level
03:09 State Space Models (SSMs) – more detail
05:45 Discretization step in SSMs
08:14 SSMs are fast! Here is why.
09:55 SSM training: Convolution trick
12:01 Selective SSMs
15:44 MAMBA Architecture
17:57 Mamba results
20:15 Building on Mamba
21:00 Do RNNs have a comeback?
21:42 AICoffeeBreak Merch

Great resources to learn about Mamba:

▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕
Join this channel to get access to perks:
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

🔗 Links:

#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research​

Scientific advising by Mara Popescu
Video editing: Nils Trost
Music 🎵 : Sunny Days – Anno Domini Beats

[drummatick]

I've a question. Given that SSMs are entirely linear, how do they conform with universal approximation theorem? I mean a lack of non-linear activation should imply that they should be particularly bad at approximate functions, but they are not.
Am i missing something?

Also really loved the video!

[partywen]

Thanks! Looking forward to a Hyena video :)

[ShadowHarborer]

I have to give a presentation on Mamba next week and I've been waiting for this video so I could finally learn what the hell I need to talk about

[peabrane]

Thank you for the shoutout to me repo!
I later realized it was an application of a known idea "heisen sequence", which is a pretty cool way to do certain associative scan operations via cumsum

[jamescunningham]

This is exactly the level of detail I needed right now. Thank you so much!

[OlgaIvina]

Thank you very much for this thorough, well-curated, and comprehensive review of MAMBA.

[cosmic_reef_]

Hats off to you for this amazing video! Best explanation of Mamba I have seen.

[DerPylz]

I was waiting for exactly this topic! Thanks so much!

[faysoufox]

Nice video, good overview, which is what I was searching for

[ruchiradhar]

A big thanks for a comprehensive explanation of the Mamba Architecture & computations, @AICoffeeBreak!

[harumambaru]

Nice T-shirt! So excited to listen about new models!

[Emresessa]

Awesome video! I especially like the simple explanation and the visuals.

[李洛克-mu]

Thank you! This is by far the easiest-to-understand and most concise video that teaches the concepts of SSMs

[hannes]

this explanation was excellent. Thank you very much :)

[kumarivin]

Thank you so much !! you really super simplified it for any beginner level deep learner to understand

[MaJetiGizzle]

Thanks for the MAMBA video!

I always appreciate your insight on these new, influential papers! Your thoughts always pair well with a good cup of coffee. 😁☕️

[고준성-mg]

Great.
There are a lot of failed explanation or completely wrong approach about SSM and Mamba on the internet, but finally I found the exact what I want.
Thank you for the video.

[rodrigomeireles]

@AICoffeBreak, thank you for the awesome video. Very small pet peeve which had me re-check all the math. At 11:20 it would make the explanation much easier to understand if you kept x 0 indexed as that is the notation you were going for since the beginning. Also, maybe making it explicit that you're taking t = L, although this is kind of obvious. This was an awesome lecture, thank you again.

[serta]

Thank you for the great Mamba explanation

[darkswordsmith]

I'm not entirely sure on how SSM differ from RNNs, especially regarding how attention is being used. Theres still the bottleneck of h_t to h_{t+1} between time steps, which was one of the motivations for the attention layer -- so that information in one part of the sequence doesn't have to be squeezed before computation with information from another part of the sequence.
Is the main innovation from RNN to SSM the fixed delta, A, B, C formulation such that the training can be done in parallel for all time steps?