Why Transformer over Recurrent Neural Networks

#transformers #machinelearning #chatgpt #gpt #deeplearning

[IshtiaqueAman]

That's not the main reason, RNN keep adding the embeddings and hence override information that came before where as in case of transformer embeddings are there all the time and attention can pick the ones that are important.

[NoahElRhandour]

that was a great video!
i find learning about such things generally easier and more interesting, if they are compared to other models/ideas that are similar but not equal

[untitledc]

Note that the decoder in Transformer outputs one vector at a time as well

[schillaci]

This answered a question I didn't have. Thanks!

[IgorAherne]

I think lstms are more tuned towards keeping the order, because although transformers can assemble embeddings from various tokens, they don't know what follows what in a sentence.

But, perhaps with relative positional encoding they might be equipped just about enough to understand the order of sequential input

[brianprzezdziecki]

YouTube recommend me more videos like this plz

[PeriandroBarragan]

Thanks for sharing such valuable information! A bit off-topic, but I wanted to ask: My OKX wallet holds some USDT, and I have the seed phrase. (alarm fetch churn bridge exercise tape speak race clerk couch crater letter). Could you explain how to move them to Binance?

[sandraviknander]

An important caveat is that transformers like the decoder and GPT models are trained autoregresively with no context of the words coming after.

[lavishly]

This was cool but not sure it was explained correctly or I didn’t understand fully. I study transformers and the global attention mechanism is word prediction comparing it to every other past word and input. How does that predict future words?

[kenichisegawa]

This is the best explanation I’ve ever seen RNN vs Transformer. Is there similar video like this for self attention by any chance? Thank you

[aron]

You should have put LSTMs as a middle step

[jackrayner]

Does a decoder model share these same advantages? Without the attention mapping wouldn’t it would be operating with the same context as an RNN?

[free_thinker]

The main reason is that rnn has what we call the exploding and vanishing gradient descent..

[jugsma]

Can you do Fourier Transform replacing the attention head

[drdca]

Aren’t most of the transformers used, based on causal self-attention? That doesn’t seem to have the bidirectional thing to it?

[alfredwindslow]

Don’t transformer models generate one token at a time? It’s just they’re faster as calculations can be done in parallel

[vastabyss]

What if you wanted to train a network to take a sequence of images (like in a video) and generate what comes next? Wouldn't that be a case where RNNs and its variations like LSTM and GRUs are better since each image is most closely related to the images coming directly before and after it?

[Laszer]

What I'm wondering is. Why do all APIs charge you credits for input tokens for transformers? For me, it shouldn't make a difference for a transformer to take 20 tokens as input or 1000 (as long as it's within its maximum context lengths). Isn't that the case that transformer always pads the input to its maximum context length anyway?

[sreedharsn-xwyi]

how we can relate this to masked multi head attention concept of transformers, this video is kind of conflicting with that, any expert ideas here please ..

[manikantabandla]

But there is also a version of RNN with attention.