How Minecraft AI ACTUALLY works

As soon as I saw AI Minecraft, I knew I had to dig in. I had no idea just how deep I'd end up going.

SOURCES

S/O Ph4seon3 for the awesome edit 🙏

[atteala-lahti]

Oh my god. It's unlimited games but no games.

[innerbytes]

It's like playing games in lucid dream. Works pretty similar there.

[ToastedTheDev]

never thought "The children yearn for the mines" would come out of theo's mouth

[TakanashiYuuji]

This reminds me of traffic in GTA5. It's only there when you look at it, turn away and it's gone.

[magicalhobo]

I can't believe theo called trapdoors "wood block things", I'm unsubscribing

[alexholker]

AI does have some good use cases for gaming. AI denoising is already used in other software to cheat on computationally expensive rendering like raycasting, and the same could be done for games to render the most expensive things at a lower resolution and use AI to guess what they should look like when upscaled. Using generative AI for text-to-speech would also allow NPCs to address your custom character by name and not as "hero, " "champion" or other placeholders, and it could be done ahead of time and the resulting audio file stored until you need it.

But for most things it sucks. You're always going to produce more optimised code working from first principles than what AI reverse-engineers by observation, and optimisation is essential if you want the highest framerate on the cheapest hardware.

[allideni]

27:37 I fail to see how this would be used to pump out tons of shitty licensed games. Like, you need video data of the game to train this thing. So they're gonna make shitty licensed games, and then sell AI models based off the shitty licensed games instead of the shitty licensed games themselves?

[hankth]

These AI games are eerily dreamlike. It weirds me out how similar a computer's attempts at simulating reality are to my sleeping brain's.

Makes me wonder how different these AI models would be if we could somehow imbue them with object permanence.

[wtfcactus]

really offtopic, but what does !gi do at 35:48 ? i tried it in hopes it would automatically open google images like yours which shouldve been the casa maybe, i just want to know what extension you use, or what does it

[okunamayanad]

6:00 thats called occulusion culling and you as a game dev generally have to configure that yourself.

for example: (not sure if this is still like this but) lethal company used to render the entire scene without doing occlusion culling. especially in buildings it means 150fps on a 4080 even with that low resolution
you can use nvidia nsight to capture the gpu api calls a game makes and reconstruct the frame in the future

[evanev]

Oh my god it took 40 minutes but finally a good theo take. As a game developer, this is so far beyond the exact opposite of everything I care about in game development. Manufactured slop at the lowest cost to make the number go up, using finite resources and talent that could actually solve problems if wielded by companies with more political interest than making money.


I see the future that AI VC wants, and I am disheartened. I see the future that FAANG want and I am afraid.

[trstmeimadctr]

It replicates the feeling of a dream better than anything I've ever seen while awake

[telephonedude]

"The children yearn for the mines" - Theo Browne, 2024

[consciouscode]

Two more papers down the line isn't just improving temporal coherency and hallucinations, it's training a foundation model which acts like a neutral physics engine and has human-tunable inputs ala Unity. Theoretically they could train a small LLM to read and interpret pseudocode (more consistent and less ambiguous than plain English) to adjust the world it's simulating, or give it a virtual filesystem to pull assets from. I also see this as very important research into infinite context using machine-addressable memory because it's _necessary_ to improve, which has applications to many other ML problems.

[roskispalo]

I actually thought this would be an amazing "minigame" for a game like GTA where it activates with drugs or alcohol

[MarcelRobitaille]

Are you thinking of xkcd 1495 for the Christmas timer server reboot thing?

[SanderCokart]

So this is a simulation for being high on shrooms xD

[codeChuck]

The idea of generative model creating a realtime video based on your prompt with no costs other than spinning the fan on your laptop faster HAS POTENTIAL!
I do not like that generative models require money for simple image prompts.
I would like it better to generate videos and pictures I wanted with permutations I wanted on my pc at no cost.

[dxniner]

would’ve been so funnily ironic if you had that ai code generator as sponsor for this video

[mr.rabbit]

33:59 Soooo AI frame inference is supposed to be an answer to costs of rendering "Hyper Realism" (Ray tracing)..??? 🤨
Am I the only one seeing the flaw in this idea?