GPT 5 is All About Data

Drawing upon 6 academic papers, interview snippets, possible leaks, and my own extensive research, I put together everything we might know about GPT 5: what will determine its IQ, its timeline and its impact on the job market and beyond.

Starting with an insider interview on the names of GPT models, such as GPT 4 and GPT 5, then looking into the clearest hint that GPT 4 is inside Bing. Next, I briefly cover reports of a leak about GPT 5 and discuss the scale of GPUs require to train it, touching on the upgrade form A100 to H100 GPUs.

Then the DeepMind paper that changed everything, focusing LLM research on data rather than parameter count. I go over a lesswrong post about that paper's 'wild implications'. And then the key paper: 'Will We Run Out of Data'. This encapsulates the key dynamic that will either propel or bottleneck GPT and other LLM improvements.

Next, I examine a different take, that perhaps data is already limited and caused the Sydney model of Bing. This opens up to a discussion on the data behind these models and why Big Tech is so unforthcoming about where it originates. Could a new legal war be brewing?

I then cover 4 of the ways these models may improve even without data augmentation, such as Automatic Chain of Thought, high quality data extraction, tool training, including Wolfram Alpha, retraining on existing data sets, artificial data generation and more.

We take a quick look at Sam Altman's timelines and host of Big Bench benchmarks that they may impact, such as reading comprehension, critical reasoning, logic, physics and Math. I address Altman's quote about timelines being delayed by alignment and safety and finally, Altman's comments on AGI and how they pertain to GPT 5.

[cytroyd]

I used to wait for GTA 5. Now I wait for GPT 5.

[califresh]

For fucks sake…this is just UNBELIEVABLE QUALITY, I can’t believe you give these insights out for free. Thank you for what you do.

[MIKEASHE]

3 things about your videos I really appreciate: 1) your clear explanations 2) ~15min bite size chunks 3) that you cite your sources on the video and description. Thank you!

[jarrod]

So, if you think about it, stopping at Sept 2021 might be because of the significant increase in AI generated content, meaning they didn't want to pollute the new AI with AI Content Generated by the Old AI to prevent Reinforcing negative parameters.

[harrypapageorgiou]

Quality information, no clickbait balderdash, just a breakdown of academic papers. This is a great metareview channel!

[JazevoAudiosurf]

1. H100 successor comes out at GTC this month, H100 price drops
2. Mega caps build H100/H100 successor farms
3. Train the models on 10+ epochs cuz who cares about energy cost when you have billions of free cash flow
4. CoT as shown in some papers vastly outperforms vanilla models, perhaps forward-forward and other algos improve performance
5. Multimodal and models connected to internet
6. Models do science and self improve

[GarryKnight]

Seems like there's no shortage of people saying, "This will change *everything*!" but few people willing to conjecture on how. You've gone quite a way in this direction, so thanks. Some of the obvious things are obvious right now, but I'm guessing that quite a few of the big changes will only be known when they happen.

[etunimenisukunimeni]

Super interesting and engaging video like always! Thanks 👍 I follow many AI focused channels, but this one takes the cake. Concise, interesting, simple and clear - and one of the few youtubers who seem to actually do their research :)

[solaawodiya]

Awesome quality content Philip. It's mind blowing on how exponential the growth of AI is.

[gabrielfernandez]

I just finished watching the video. Your videos are very high quality and I hope you get very famous soon! 😁

[yoseidman]

Thank you for all of your hard work. You are doing an amazing service to all of us.

[DanielSeacrest]

This video is really high quality, and makes learning about how these models could improve so much easier, thanks!

[EnemyOfEldar]

I remember the CGPGrey video "Digital Aristotle" and I couldn't wait and it's finally going to happen. When kids have AI to teach them, no one will be left behind.

[praveen]

Thank you for putting out such a well-researched, high quality summary! You've earned yourself a new subscriber.

[jeanchindeko]

12:17 the training cost is a 1 time cost compared to the cost of running those big model over 150B parameters.
If the model is smaller and more performant it’s also cheaper to run inferences

[GabriellaSannino]

Brilliant... I found this video on an Italian thread in Linkedin, and with over 2.8k likes in 11 days, you seem to be on fire!!

[josephgenereux]

Great video! A major gap that I couldn’t stop thinking about wrt to training data was the scope was locked down purely to the tokenization of written content. 1 sensory input if you will. I like what Andrej Karpathy mentioned (can be seen on his Lex F interview): “the path to AGI is multi-modal”. If we tokenize the senses of sound and vision (via tokenization of audio and video data, both live and recorded) and explore other sensory inputs, how would that change the curve? Surely becoming more human like would include more human like sensory input. The literature is riddled with clues here. Food for thought.

Excellent video. Your content is always worth a watch. Very high quality.

[abdifatahnadir]

Keep creating and sharing as many videos as you can!

[timokreuzer]

2020: "Oh, you got fired? Learn to code!"
2028: "Oh, you got fired? Learn to plumb!"

[dontmindmejustwatching]

OpenAI: we might release GPT 4 someday, but don't expect much.
Content creators: GPT 55 confirmed