The Narrated Transformer Language Model

AI/ML has been witnessing a rapid acceleration in model improvement in the last few years. The majority of the state-of-the-art models in the field are based on the Transformer architecture. Examples include models like BERT (which when applied to Google Search, resulted in what Google calls "one of the biggest leaps forward in the history of Search") and OpenAI's GPT2 and GPT3 (which are able to generate coherent text and essays).

This video by the author of the popular "Illustrated Transformer" guide will introduce the Transformer architecture and its various applications. This is a visual presentation accessible to people with various levels of ML experience.

Intro (0:00)
The Architecture of the Transformer (4:18)
Model Training (7:11)
Transformer LM Component 1: FFNN (10:01)
Transformer LM Component 2: Self-Attention(12:27)
Tokenization: Words to Token Ids (14:59)
Embedding: Breathe meaning into tokens (19:42)
Projecting the Output: Turning Computation into Language (24:11)
Final Note: Visualizing Probabilities (25:51)

The Illustrated Transformer:

Simple transformer language model notebook:

Philosophers On GPT-3 (updated with replies by GPT-3):
-----

More videos by Jay:
Jay's Visual Intro to AI

How GPT-3 Works - Easily Explained with Animations

[parthchokhra]

Your blog on Illustrated Transformer was my intro to Deep Learning with NLP. Thanks for the amazing contributions for the community.

[andresjvazquez]

Dear Teacher Alammar, thanks to this video I was able to accepted into BYU lab as an external researcher (even though I didn’t finish college) and have been invited by my professor to participate with the lab in CASP15 . You really changed the course of my life by demystifying such complex topics for non traditional learners like me . I’m eternally in your debt

[ans]

The Illustrated Transformer blog is a masterpiece!

[Roshan-xdtl]

Your ability to explain and breakdown complex topics into simpler and intuitive sections is legendary. Thank you for your contribution!

[ayush]

I remember Seeing your Transformer's Blog Jay.. It was legendary!! Was referred to by other youtubers as well... And thanks a lot for the wonderful explanation as well!

[diogo.magalhaes]

Jay, as a PhD student, I'm a fan of your ability to explain complex topics, in a very simple, illustrated and didactic way! I always recommend your ' illustrated' posts to my colleagues. Thanks again for this great video, keep up the good work!

[tachyon]

It would nice to have a step by step walkthrough of the training process. And why each of those steps makes sense intuitively.

[bighit]

you have a gift for explaining complex materials... many other technical talks assumes the audience is very knowledgeable and are attending the session just for networking

[quietkael]

Thank you so much for all the tireless work you do for us visual learners out there! I’m looking forward to videos where you get into your excellent visualizations of the underlying matrix operations. Your visual abstractions both at the flow chart level and matrix/vector level have really shaped my mental model for what I think about when I’m engineering models. I’m so grateful and so excited to see what you come out with next (this library you hint at looks wonderful!)

[kalinda]

A phenomenal extension of your blog post. Commenting for that bump in the recommendation algorithm!

[nileshkikle]

Outstanding job demystifying the inner working details of the Transformer model architecture! All the illustrations and animations for the inference working are awesome. Thank you for taking all the time and sharing your understanding with all of us. Kudos! 👍

[maruthiprasad]

Amazing explanation, my search to understand the transformers ended here, you done the wonderful job, thank you so much for the simplest explanation I ever seen.

[drtariqahmadphd]

Never been more excited by a YouTuber channel than when I saw this guy had a channel.

[goelnikhils]

I haven't see such a clear explanation of Transformers and Decoder LM Models, Amazing Work Jay

[curiouspie]

One of the most comprehensive video and blog overviews of Transformers I've seen. Thank you. 🙏

Just a personal comment on the format of the videos: I, personally, find that constant change of scene (like in "The architecture of the transformer" section) where the camera changes constantly showing you and then showing the computer screen and then back to you, is extremely annoying.

The content of the video itself was informative.

[jemmaj]

this is amazing. One thing I didn't understand is the matrix, how it is generated and used in the processing to return the probability (how "the" turns into a big array of inputs)

[kazimafzal]

You sir are an amazing teacher! I'm absolutely flabbergasted by how well you've explained, to think its all mathematics at the end of the day! Thank you for taking the time to put together such a concise yet complete guide to transformers!

[rsilveira]

Nice collection of albuns man! Miles Davis, Radiohead, John Coltrane, very classy! 👏👏👏

[JimBob-lqdb]

Thank you for this great explanation. Visualize, visualize, visualize, the best way to undestand how it works.