MIT 6.S191 (2022): Recurrent Neural Networks and Transformers

MIT Introduction to Deep Learning 6.S191: Lecture 2
Recurrent Neural Networks
Lecturer: Ava Soleimany
January 2022

Lecture Outline
0:00​ - Introduction
1:59​ - Sequence modeling
4:16​ - Neurons with recurrence
10:09 - Recurrent neural networks
11:42​ - RNN intuition
14:44​ - Unfolding RNNs
16:43 - RNNs from scratch
19:49 - Design criteria for sequential modeling
21:00 - Word prediction example
27:49​ - Backpropagation through time
30:02 - Gradient issues
33:53​ - Long short term memory (LSTM)
35:35​ - RNN applications
40:22 - Attention fundamentals
43:12 - Intuition of attention
44:53 - Attention and search relationship
47:16 - Learning attention with neural networks
54:52 - Scaling attention and applications
56:09 - Summary
Subscribe to stay up to date with new deep learning lectures at MIT, or follow us @MITDeepLearning on Twitter and Instagram to stay fully-connected!!

[andreas.karatzas]

I'm still trying to figure out how did you manage to perfectly describe the logic behind attention mechanisms in 10 minutes ...

[jzhuo]

This is so well explained - thanks a lot

[jacktrainer]

Everything that comes out of MIT is pure gold. You'd think that the concepts would be described at a high, inaccessible level, but that's not the case. The lectures are student friendly & homeworks are challenging and doable.

[chiranjeevisagi]

Thank you to Alexander Amini and Ava soleimany for making this course accessible to everyone, which otherwise is a distant dream for many people like myself to learn such high quality content.

[SteveSperandeo]

Who needs GPT-3 when we have Ava? Amazingly clear, succinct, and enjoyable presentation. Thank you Ava!

[mohammadalaaelghamry]

Just amazing how well those two lectures are layed out, structured and explained, nothing comes close to them in my experience so far, thank you so much Alexander and Ava, heading for the first lab now.

[laminsesay]

if you are watching, learning and practicing this video, you have be granted a visa to the future. Alexander Amini, Ava Solemany and the rest of the team thanks. you guy are amazing

[SinkingPoint]

This is by far the best explanation of attention that I've seen. It definitely deserves its own video. Maybe a video on transformers that covers attention and some more detail on the other components of the architecture?

[tantzer]

Excellent lecture. Very well designed, clear, intuitive, well balanced. A lot was accomplished in one hour! I learned a lot.

[arnavraina]

Feels like I'm waiting for a much awaited movie trailer! This is quality.

[robertoooooooooo]

this is genius. This lecture is pure gold. Such difficult concepts like transformers explained in a 15 minutes seems to be impossible but she did it. Thank you MIT!

[ajaytaneja]

"Attention Is All you Need" - The intuition of Query, Key and Value is one of the best from what I've read or watched (in other courses) until now....Excellent job Ava Soleimany, thank you

[helloansuman]

Very fast videos. Need to slow down and explain key concepts clearly. Otherwise it's like a sweet story.

[caiomar]

I love these series! Thank you for sharing the knowledge! I am listening to very word! Now I am getting Instagram ads for MIT Full AI course for the hefty price of $3300 USD, I wish I could afford it ;/

[ImtithalSaeed]

unable to describe how amazing is this ... thank you Ava

[ShaidaMuhammad]

Finally, I understood the self attention mechanism completely.

[dianakapralova]

Thanks for detailed explanations. Especially, attention!And finally attention all that we need and additionally understand thanks to you:-)

[hyewoncho]

This is definitely the best video for describing attention mechanisms and the logic behind them. Many videos only try to review as it is written in the paper. Thank you so much! It really helped me a lot to get the attention even more clearly!

[asokakarunananda]

Excellent presentation on the transition from RNN to Attention-based Transformer networks. Thank you

[SeshaB]

Excellent explanation! This is perhaps the best description about the roots of the attention mechanism, and the intuition behind it. People who follow the route of CNNs -> GANs -> ViTs in their deep learning journey have trouble in understanding the self-attention (without having much knowledge about RNNs). This is like an excellent "bridge" video that fills all the gaps! Great effort by Ava!