Coding a ChatGPT Like Transformer From Scratch in PyTorch

In this StatQuest we walk through the code required to code your own ChatGPT like Transformer in PyTorch and we do it one step at a time, with every little detail clearly explained.

NOTE: This StatQuest assumes that you are already familiar with the concepts behind...

If you'd like to support StatQuest, please consider...
...or...

...buying my book, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...

...or just donating to StatQuest!
venmo: @JoshStarmer

Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:

0:00 Awesome song and introduction
1:12 Loading the modules
2:04 Creating the training dataset
6:17 Coding Position Encoding
14:09 Coding Attention
21:04 Coding a Decoder-Only Transformer
26:39 Running the model (untrained)
29:18 Training and using the model

#StatQuest #PyTorch #chatgpt

[thebirdhasbeencharged]

Can't imagine the work that goes into this, writing the code, making diagrams, recording, editing and voice over, you're the goat big J.

[techproductowner]

You will be rememberd for next 1000 years in the history of Statistics and Data Science, You should be named as "Father of Applied Statistics & Machine Learning ", Pls thumbs up if you are with me

[jahanzebnaeem]

HUGE RESPECT for all the work you put into your videos

[hewas]

Hey Josh, you know what? I used to watch your videos explaining the key ingredients of statistics EVERY DAY in 2020~2021 when I was a freshman. Whatever I click among your videos, it was always the first time for me to learn it. I knew nothing. But I still remember what concept you dealt with in videos and how you explained them.

Fortunately now I work as an AI researcher - it's been a year already - although I am a 3rd grade student. You suddenly came to my mind so I've just taken a look at your channel for the first time in a long time. This time I've already knew about all of what you explain in videos. It feels really weird. Everything is all thanks to you and still your explanations are clear, well-visualized and awesome. You are such a big help to the newbies of statistics and machine/deep learning. Always love your works. Please keep it going!!! 🔥

[n.h.son]

You said this was going to come out at the end of May. And I’ve been waiting for this for 2 months. Finally, it’s out 😂

[Opczak]

Josh, I want to express my sincerest gratitude. I have been following your videos for years and they have been becoming increasingly more important for my study and career path. You are a hero.

[muhammadikram]

sir you deserved millions of views on your YouTube ❤❤🎉

[progameryt]

Incredible video, Josh! Love your content. Can you please make a video on diffusion models?

[TalkOfWang]

It is party time! Thanks for uploading!

[bayoudata]

Cool, learn a lot from all of your videos Josh! 🤯

[ramwisc]

Wow - have been waiting for this one! Now that I've wrapped my head around word embeddings, time to code this one up! Thank you @statquest!

[akshaygs]

It had been sometime since i watched your video. Very good video as always 🎉🎉

[abhinavsb]

100/100 🔥when i search for an explanation video on youtube this is what i expect🔥

[Sravdar]

AMAZING VIDEOS. Watched all of your nn playlist in 3 days. And now reaching the end i have some questions. One is what are the future planned videos? And two is how do you select activation functions? In fact a video where you create custom models for for different problems and explaining "why to use this" would be great. No need to explain math or programing needed for that.

Thank you for all of these videos!

[sillypoint]

This video's amazing man. Not just this one but every video of yours. Before I began actually learning Machine Learning I used to watch your videos jus for fun and trust me, it had taught me a lot. Thanks for your amazing teaching :) with love from India ❤

[gvascons]

Great and very didactic as usual, Josh!! Definitely going to wrap my head around this for a while and try a few tweaks! Do you plan on eventually also discussing other non-NLP topics like GANs and Diffusion Models?

[Brad-qwte]

I’ve been trying to make a Neural Network in c++ for like a month now. I was trying to just use 3b1b’s videos but they wernt good enough. But then I found your videos and I’m getting really close to being able to finish the back propagation algorithm.

When I started I thought it would look good on my resume but now I’m thinking nobody will care but I’m in too deep to quit

[iqra]

Amazing explanation 🎉❤ you are the best 😊

[datasciencepassions]

God Bless You for the great work you do! Thank you so much

[jorgesanabria]

This will be awesome. I am trying to learn the math behind transformers and PyTorch so hopefully this helps give me some intuition