Stanford CS25: V4 I Overview of Transformers

April 4, 2024

[ilm_yanfa]

Can't believe, ... Just today, we started the part about LSTM and transformers in my ML course, and here it comes
Thank you guys !

[Drazcmd]

Very cool! Thanks for posting this publicly, it's really awesome to be able to audit the course :)

[fatemehmousavi]

Awesome, thank you Stanford online for sharing these amazing video series

[benjaminy.]

Hello Everyone! Thank you very much for uploading these materials. Cheers

[marcinkrupinski]

AMazing stuff! Thank you for publishing this valuable material!

[JJGhostHunters]

I recently started to explore using transformers for timeseries classification as opposed to NLP. Very excited about this content!

[mjavadrajabi]

Great!! Finally It's time for CS25 V4🔥

[lebesguegilmar]

Thanks for sharing this course and palestry Staford. Congratulations . Here the Brazil

[styfeng]

it's finally released! hope y'all enjoy(ed) the lecture 😁

[GeorgeMonsour]

I want to know more about 'filters.' Are they human or computer processes or mathematical models? The filters are a reflection, I'd like to understand more about. I hope they are not an inflection, that would be an unconscious pathway.

This is a really sweet dip into the currency of knowledge and these students are to be commended however, in the common world there is a tendency developing towards a 'tower of babel'.

Greed may have an influence that we must be wary of. I heard some warnings in the presentation that consider this tendency.

I'm impressed by these students. I hope they aren't influenced by the silo system of capitalism and that they remain at the front of the generalization and commonality needed to keep bad actors off the playing field.

[GerardSans]

Be careful using anthropomorphic language when talking about LLMs. Eg: thoughts, ideas, reasoning. Transformers don’t “reason” or have “thoughts” or even “knowledge”. They extract existing patterns in the training data and use stochastic distributions to generate outputs.

[TV]

future artificial intelligence
i was into talk this
probability challenge
Gemini ai talking ability rapid talk i suppose so
it's splendid

[egonkirchof]

In summary, Transformers mean using tons of weight matrixes, leading to way better results.

[Anbu_Sampath]

it would be great if CS25: V4 created another playlist in youtube.

[ramsever]

what is said in 13:47 is incorrect.
Large language models like ChatGPT or other state-of-the-art language models do not only have a decoder in their architecture. They employ the standard transformer encoder-decoder architecture. The transformer architecture used in these large language models consists of two main components:
The Encoder:
This encodes the input sequence (prompt, instructions, etc.) into vector representations.
It uses self-attention mechanisms to capture contextual information within the input sequence.
The Decoder:
This takes in the encoded representations from the encoder.
It generates the output sequence (text) in an autoregressive manner, one token at a time.
It uses self-attention over the already generated output, as well as cross-attention over the encoder's output, to predict the next token.
So both the encoder and decoder are critical components. The encoder allows understanding and representing the input, while the decoder enables powerful sequence generation capabilities by predictively modeling one token at a time while attending to the encoder representations and past output.
Having only a decoder without an encoder would mean the model can generate text but not condition on or understand any input instructions/prompts. This would severely limit its capabilities.
The transformer's encoder-decoder design, with each component's self-attention and cross-attention, is what allows large language models to understand inputs flexibly and then generate relevant, coherent, and contextual outputs. Both components are indispensable for their impressive language abilities.

[ummnine]

i thought it will be extensively detailed lecture on transformers to teach people exactly how it works, but this was nothing more then modern ai news and very high level explanation of the news, very
disappointing

[laalbujhakkar]

Stanford's struggles with microphones continue.

[si]

This is not what I expected. What a complete terrible explanation. I was expecting a complete history of Transformers. The fall of the Deception's or how Optimus Prime came to be. A very misleading title indeed.