MIT 6.S094: Recurrent Neural Networks for Steering Through Time

This is lecture 4 of course 6.S094: Deep Learning for Self-Driving Cars taught in Winter 2017.

INFO:

Links to individual lecture videos for the course:

Lecture 1: Introduction to Deep Learning and Self-Driving Cars

Lecture 2: Deep Reinforcement Learning for Motion Planning

Lecture 3: Convolutional Neural Networks for End-to-End Learning of the Driving Task

Lecture 4: Recurrent Neural Networks for Steering through Time

Lecture 5: Deep Learning for Human-Centered Semi-Autonomous Vehicles

CONNECT:
- If you enjoyed this video, please subscribe to this channel.

[sewellrw]

I've been having these videos come up as recommendations to watch on youtube. Lex is so good at teaching these topics that can get a bit complicated. I wish I had more professors like Lex in college for courses that had difficult to understand concepts. He really breaks things down slowly, and explains things in a way that people can understand.

[stefanfaiciuc]

Thanks so much for this, Lex. Your lecture was how I finally understood how RNNs work and it helped me to successfully complete my university thesis back in 2017. It's funny how I came across you again through Joe Rogan and your podcast and figured it's the same dude that helped me through college. Hope you get to be the one that builds robots better than anybody else in the world.

[abdulelahalkhoraif]

I have to say you are very talented for teaching very complex topics. Thank you so much MIT for choosing such a brilliant presenter.

[MasterAufBauer]

About the Udacity Challenge: It is very likely that none of these winning models will be able to steer a car. What they are really good at is predicting the steering angle depending on the last few frames. Actually you may be able to compute that by using the difference image of the last two frames without any network and achieve almost the same performance as the winning teams. As Nvidia mentioned in their paper End to End Learning for Self-Driving Cars without learning error correction the car will just leave the center of the road and the network has no idea how to correct that because the situation is not known from the training data. My point is steering a car correctly on a real road and the prediction of steering angles based on a video are two different challenges.

[anthonyarthurmensah]

Ive been following your videos but never knew you are/were a tutor/lecturer... I am going to enjoy this.

[Gannicus]

Some, hopefully helpful for the audience, remarks:
1. You need a lot of data. Depends. A lot of unlabeled data helps — to model the world. Then u need very little supervised data. Easy problems require little data. Hard or badly defined tasks require a lot of data. You can always pick an easier to solve proxy objective and use data augmentation.
2. rnns are dynamic length. Hard Set sequence lengths are for speed since:
sentences come at different lengths. So u cant create batches, unless you set a hard sequence length and then train same length sentences together in a batch, or fill up sentences that are too short by padding.
If you batch sentence you can compute on them in parallel.

Now of you are trying to predict relations between consecutive sentences, batching/ parallelization would not update the weights after each sentence, but on all of them at once — making it near impossible to learn inter (between) sentence relations but allowing the net to learn intra (within) sentence relations.

Tip: read karparthys blog on rnns not the Colah one. Karpathys is more detailed allowing you to really grasp what an rnn does. An lstm is „just“ and rnn with attention/gating.

Hope this helps, even if some concepts are very high level.

[vman]

Great video, but I wish there was more math and a more thorough explanation of BPTT and the vanishing gradient problem.

[matiasiribarren]

"It's producing something that sounds like words...That could do this lecture for me. I wish..." 1:02:42 Oh Lex would rather be researching xD.

[anthonybyne]

I've never listened to anyone before without understanding anything at all. It's fascinating for me watching with zero understanding. I'm literally just listening to his words... 😂

[chrysr]

Ooh I recognize what's on the blackboard! It's the spherical coordinate system...

[niazhimselfangels]

Hi Lex, thank you for these lectures! Would you be uploading the guest lectures as well? There isn't any mention of them in the course home page now .

[AlaveNei]

I find the subject content, 'Recurrent Neural Networks for Steering Through Time', very well presented.
An interactive class environment that was improvised self-learning discussion topics were explained.

A side note: Keep your ‘walking through the abstract’ as your own understanding, which I believe can be really useful to learners, and thank you for keeping it up :)

[john-paulglotzer]

Really great video. Thanks! Quick question... At 1:13:34, the team used a CNN to create a distributed representation of each frame, and then they use this as the input to the RNN. Was this just a generic CNN trained on completely different types of images? Or did they train a new one using the driving images? If the latter, what target variable would they use to train it? Thanks!

[nemis]

This gave me so much understanding. Thank you for uploading!

[klrshak]

1:14:17 Who else thought their battery was low paused the video to check their battery😂🤣😂🤣

[Constantinesis]

Hello, thanks for uploading these lectures! Can LSTM networks integrate symbolic constructs in natural language learning? Can it help computers understand the relationship between language structure and real world? For example if I ask "Why is only raining outside? " It should know that the roof stops the rain falling inside. I have a feeling that we are mostly teaching the algorithm to interact with us, in some kind of smart language simulation but at it's core it doesn't really understand the meaning and relationships between words. Do you know some online references towards this?

[JamesNortonSound]

Am I the only person who thought that the video compression makes his shadow look like a low resolution shadow map...? Awesome content, great for getting into ML!

A quick question regarding LSTM's, why do we need a separate way of saying 'this information isn't important, I don't want to update my weights'. Doesn't gradient descent already take care of this? That is, if we happen to see a feature that is unimportant, won't we compute low gradients, thus telling us we don't need to move the weight by much? Why doesn't that work here?

[quintrankid]

Parameter tuning can't be taught? But it can be learned? I wonder if that would be a useful thing to apply ML to?

[mikashen]

Hey, Lex. Really great video! But as English is not my mother tongue, sometimes it's difficult to understand the video very well, it would be nice if you can turn on the cc subtitle options, thanks!

[funduk]

I think before introducing the backprop it is a good idea to start with the forward mode