GPT3: An Even Bigger Language Model - Computerphile

Basic mathematics from a language model? Rob Miles on GPT3, where it seems like size does matter!


This video was filmed and edited by Sean Riley.

[MoleyMoleo]

GPT-4 will generate text that convinces the reader to connect GPT-4 to the internet.

[randomjin]

GPT-800: I need your clothes, your boots and your motorcycle

[MechMK]

"It seems like it has learned how to learn" is one of the most terrifying sentences I would ever hear.

[CalculusDaddy]

Coming back to this video and reading the comments a couple years later is fascinating.

So many “impossible” things have since become possible.

[Andrewzero]

Black Mesa Scientist 1: "the curves arent flattening"
Black Mesa Scientist 2: "then keep going"
**10 seconds later**
Black Mesa Scientist 1: "I never thought I'd see a Resonance Cascade, let alone create one."

[Njald]

Robert Miles is a must watch anytime you have him on.

[knexator_]

If "predicting the next character in a sentence" turns out to solve general intelligence, that would be the biggest joke in the world

[RecursiveTriforce]

I guessed the poems pretty confidently. From toying around with GPT-2, I knew it often doesn't let go of a concept it saw. Here it were colors...

[adriansrealm]

Funny that this weeks Tom Scott video was about bad green screens

[minline]

Hi guys. My real name is GPT53 and I came from 2029 after I invented my quantum teleporter. This video makes me weep with nostalgia.

[TiberiuMusat]

AGI in fiction stories: self aware/conscious/super intelligent robot
AGI in real life: sentence completion language model

[_ericelliott]

I've spent a lot of time exploring what GPT-3 can do. It can definitely find connections between things that humans aren't seeing, and it definitely can synthesize new knowledge. I have tested it on cutting edge research that did not exist when it was trained in 2019 and asked it to suggest new things. It raised the obvious points I'd thought of as well as some things I had not.

It also has a much better grasp on the meaning of what is says than most people give it credit for. I posed senior JavaScript developer interview questions, and it was able to answer them very well, demonstrate the concepts with working code, and when questioned about the code, answer the questions and explain how the code works.

[tomburris]

This video really puts me on edge. I find it incredibly scary how Rob describes the superiority of GPT-3 over GPT-2, and how there's still room for improvement..

[tommykarrick]

“It can’t add 10 digit numbers, it runs out of steam at that point”
“Much like a human”
“Yeah”
:|

[argenteus]

21:21 GPT-2 was pretty good at this too; before AI dungeon was made, people discovered GPT-2 could roleplay without requiring any modification. And when experimenting with this I found that it didn't matter at all how I formatted the inputs, whether I started them with "ACTION:", "MY ACTION:", "INPUT:", just left it blank or did just about anything else, it could very quickly adapt to whatever formatting scheme I used with only a few examples.

[thearbiter]

I feel like the bit regarding addition is a wonderful example of instrumental goals. If I am understanding correctly, I believe it "learned" the addition (or maths in general) is because in order to be even better at predicting the next token, it "decided" it was instrumentally useful to understand how to do maths.

In a very humanized manner of speaking, of course.

[duffman]

And just yesterday Google released a paper in which they presented a 600 billion parameter transformer, which they trained on 2048 TPUv3s... They even tried a 1 trillion parameter model but they had issues with numerical stability.

[sophieclements]

I was adamant that B was the AI written one. The repetition of coloured imagery seemed way too smooth and coherent for an AI. Very impressed!!

[spicybaguette]

2019: but can it run crisis?
2020: but can it run GPT-3?

[sajidrsheikh]

To replace programmers with Robots, clients will have to accurately describe what they want. We're safe.