GPT2 Explained!

This video explores the GPT-2 paper "Language Models are Unsupervised Multitask Learners". The paper has this title because their experiments show how massive language models trained on massive datasets can perform tasks like Question Answering and Translation by carefully formatting them as language modeling inputs.

Paper Links

Thanks for watching! Please Subscribe!

[connor-shorten]

1:08 GPT Background
2:35 Bigger Data!
3:30 Bigger Models!
4:15 Zero-Shot LM Transfer
5:24 Zero-Shot NLP Tasks
5:57 Question Answering, Allen Institiute Demo
7:22 Transformer Decoder
7:54 GPT-2 Translation
8:20 Larger Models, 17B Turing-NLG
9:20 Misuse: Reddit GPT2 + BERT Bot

[KeinNiemand]

In just a few years we went from GPT-2 to ChatGPT

[HKMOfficial]

Great video — concise, yet covers a lot of material I had questions about!

[Kanny]

Hi Henry! I have a question, why is it called "unsupervised"? I mean, the training is performed on prediction of the next token, so why would you consider it unsupervised?
The model learns P(x(t+1) | x(t), x(t-1)... x(0)), right?

[OccultDemonCassette]

Jesus. Is there not a GUI for this where I can just drag and drop like 2000 text files into a box and click "train"

[maksgorczynski]

I asked for a short story involving harame.
What I got however was this.

As harambe lay in a pool of his own blood, he spotted the glint of a gun. The ape cried. "How come I, man?" but it would not bear him there.

"Well—why, why not? Who are you?'

A woman, at once, asked.

"I am a slave of Master Cunha. He is my man. He holds nothing to me but my faith. I do not have any slaves, but Cunha keeps them all away from me."

He glanced at her, then at Cunha. There seemed to be a sharp tug of his hand.

"There is something else!" said he.

"No, not the one—"

"I suppose that it will be better. We shall get him a chance to give him a lesson now. But if he wants to do it I shall let you know, then I will tell her you are one."

"I am sure he shall."

The man took his hand.

"I am afraid I am not going to come but by your invitation. I have already told him you are of his own, and that you must not speak to him. Give him the opportunity to see you. He has to see you, " he ordered, after a pause.

"But I don't know, " said he. "I'm sure he understands what you are doing and knows what

[Patrick]

I had no idea that GPT existed 3 years ago.

[stormychristian]

I believe a man used this to make fake content messages that he claimed were from, it has ruined my life. Is there a way to detect the fake words? Please help

[joliver]

Quick suggestion to slow down and explain more. Seems as though you are quickly reading to us at times. Good topics though.

[grownupgaming]

2:40 "the way this dataset is constructed is a huge contribution to this paper" .... proceeds to show that "women who use vibrators both alone and with a partner experience greater" I think +3 karma was the cutoff? So all it takes is for three adolescents who really need to get outside more, for it to become groundbreaking paper-worthy knowledge. lol