Building makemore Part 5: Building a WaveNet

Links:

Supplementary links:

Chapters:
intro
00:00:00 intro
00:01:40 starter code walkthrough
00:06:56 let’s fix the learning rate plot
implementing wavenet
00:17:11 overview: WaveNet
00:19:33 dataset bump the context size to 8
00:19:55 re-running baseline code on block_size 8
00:21:36 implementing WaveNet
00:37:41 training the WaveNet: first pass
00:38:50 fixing batchnorm1d bug
00:45:21 re-training WaveNet with bug fix
00:46:07 scaling up our WaveNet
conclusions
00:46:58 experimental harness
00:47:44 WaveNet but with “dilated causal convolutions”
00:52:28 the development process of building deep neural nets
00:54:17 going forward
00:55:26 improve on my loss! how far can we improve a WaveNet on this data?

[khisrowhashimi]

I love how we are all so stressed and worried that Andrej might grow apathetic to his YouTube channel, so everyone wants to be extra supportive 😆 Really shows how awesome of a communicator he is.

[antonclaessonn]

This series is the most interesting resource for DL I've come across, being a junior ML engineer myself. To be able to watch such a knowledgeable domain expert as Andrej explaining everything in the most understandable ways is a real privilege. A million thanks for you time and effort, looking forward to the next one and hopefully many more.

[nervoushero]

As a independent deep learning undergrad student ur videos helps me a lot. Thank u andrej Never stop this series.

[GlennGasner]

I really, really appreciate you putting in the work to create these lectures. I hope you can really feel the weight of the nearly hundred thousand humans who pushed through 12 hours of lectures on this because you've made it accessible. And that's just through now. These videos are such an incredible gift. Half of the views are me because I needed to watch each so many times in order to understand what's happening, because I started from so little. Also, it's super weird how different you are from other YouTubers and yet how likable you become as a human during this series. You are doing this right, and I appreciate it.

[crayc]

Notification for a new andrej video guide feels like a new season of game of thrones just dropped at this point.

[timelapseguys]

Andrej, thanks a lot for the video! Please do not stop continuing the series. It's an honor to learn from you.

[RishikeshS-nvol]

Please don't stop making these videos, these are gold !

[ZaphodBeeblebrox]

I experimented a bit with the MLP with 1 hidden layer and managed to scale it up to your fancy hierarchical model. :)
Here is what i got:

MLP(105k parameters):
block_size = 10
emb_dim = 18
n_hidden = 500
lr = 0.1 # used the same learning rate decay as in the video
epochs =
mini_batch = 32
lambd = 1 ### added L2 regularization

seed is 42
Training error: 1.7801
Dev error: 1.9884
Test error: 1.9863 (I checked this only becouse I was worried that somehow I overfitted the dev set)

Some examples generated from the model that I kinda liked:
Angelise
Fantumrise
Bowin
Xian
Jaydan

[rajeshparekh]

Thank you so much for creating this video lecture series. Your passion for this topic comes through so vividly in your lectures. I learned so much from every lecture and especially appreciated how the lectures started from the foundational concepts and built up to the state-of-the art techniques. Thank you!

[maestbobo]

Best resource by far for this content. Please keep making more of these; I feel I'm learning a huge amount from each video.

[hintzod]

Thank you so much for these videos. I really enjoy these deep dives, things make so much more sense when you're hand coding all the functions and running through examples. It's less of a black box and more intuitive. I hope this comment will encourage you to keep this going!

[Jmelly]

I rarely comment on videos, but thank you so much for this series, Andrej. I love learning, and learn many things that interest me. The reason I say that is that I have experienced a lot tutors/educators over time. And for what it's worth, I'd like you to know you're truly gifted when it comes to your understand of AI development and communicating that understanding.

[vivekpadman]

Absolutely love this series Andrej sir... It not only teaches me stuff but gives me confidence to work even harder to share whatever I know already.. 🧡

[mipmap]

Can't wait for part 6! So clear and I can follow step by step. Thanks so much

[SmeMthersson]

Your work ethic and happy personality really move me. Respect to you, Andrej, you are great.🖖

[brittaruiters]

I love this series so much :) it has profoundly deepened my understanding of neural networks and especially backpropagation. Thank you

[aurelienmontmejat]

This is the best deep learning course I've followed! Even better than the one on Coursera. Thanks!

[WarrenLacefield]

Enjoying these video so much. To refresh most of what I've forgotten about Python and to begin playing with pytorch. Last I did this stuff myself was with C# and CNTK. Now going back to rebuild and rerun old models and data (much faster even & "better" results). Thank you.

[NarendraBME]

So far THE BEST lecture series I came across on YouTube. Along side learning the neural networks in this series, I have learned the PyTorch more than learning it by waching a PyTorch video series of 26 hrs from a youtuber.

[sakthigeek]

Learned a lot of practical tips and theoretical knowledge of why we do what we do and also the history of how Deep Learning evolved. Thanks a lot for this series. Requesting you to continue the series.