DeepMind's AI Learns To See | Two Minute Papers #263

Crypto and PayPal links are available below. Thank you very much for your generous support!
Bitcoin: 13hhmJnLEzwXgmgJN7RB6bWVdT7WkrFAHh
Ethereum: 0x002BB163DfE89B7aD0712846F1a1E53ba6136b5A
LTC: LM8AUh5bGcNgzq6HaV1jeaJrFvmKxxgiXg

The papers "Neural scene representation and rendering" and "Gaussian Material Synthesis" are available here:

We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
313V, Andrew Melnychuk, Angelos Evripiotis, Brian Gilman, Christian Ahlin, Christoph Jadanowski, Dennis Abts, Emmanuel, Eric Haddad, Esa Turkulainen, Geronimo Moralez, Kjartan Olason, Lorin Atzberger, Marten Rauschenberg, Michael Albrecht, Michael Jensen, Morten Punnerud Engelstad, Nader Shakerin, Rafael Harutyuynyan, Raul Araújo da Silva, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Thomas Krcmar, Torsten Reil, Zach Boldyga.

Károly Zsolnai-Fehér's links:

[movaxh]

This is so crazy. 10 years ago, I was researching this exact idea, and honestly couldn't do much progress. I was using classical vision and heuristics, but at the end I knew it will be neural network based solution, with learning of 3d relations, and understanding of shadows and iluminagion under partial information, to make it useful. Still there are conditions that are extremely hard to use, like reflective surfaces, uneven or colored lights, etc. People have intuitive understanding of how various things (apple, a chair, face, window, painting frame, glass cup) should look, what are they usual sizes, and that thing usually lay on top of another ones or attached to walls, not floating in space, even if the picture is from single view point one can reconstruct positions and depth and occluded parts by analogy and common sense. This algorithm is a step in good direction.

[crazyMLC]

could enable continuous movement in google street view

[MobyMotion]

I'd love to see someone port an easy to use neural renderer into Blender so people can start using it to create great looking scenes quickly. Karoly, how easy is it for a relative beginner to set up your research on their own system?

[hdef]

channels like yours are the reason I can't quit YouTube

[crazyMLC]

if this could be used for art instead of just renders or photos, then it could enable the creation of 3d models from a couple drawings. (would help a lot of content creators and might get used in fields like game development)

[The_Jaganath]

All I can think is blistering fast, super hi res VR environments could be possible with this type of 'on-the-fly' rendering :0) What a time to be alive!

[deafharp]

Love how frequently you upload these. Thanks for your hard work/research

[BjarkeDuDe]

This could be used for 3D scanning! Imagine only having to take a couple pictures of a room, and the algorithm will synthesize missing content to create a better 3D reconstruction using photogrammetry!!

[MrKohlenstoff]

I wonder if you can use this to create really weird pseudo-geometries. Like taking a bunch of unrelated pictures as input, and see what the network comes up with as geometric representation, and how it looks to view this space from different angles. It might be completely useless chaos, or it might be mindblowing. Or somewhere in between. Maybe!

[PatricioToledo]

Thanks, I love your videos. This could be applied to take 2d images from surveillance cameras and recreate 3d scenarios, whether is from a Crime Scene or to reconstruct historical footage or concerts, etc. Excellent.

"What an awesome time to be alive!!!"

[aethermass]

In the future, this could be used to speed up real-time 3D rendering. Making VR available to computationally weaker devices would be a big plus. No more wires on VR headsets!!

[helloansuman]

Very nice please carry on this series.

[LucidSurreal]

I was just imagining something like this the other day. The idea would be to have a bunch of mini drones with 360 degree cameras and have them swarm onto a real life scene. Then you could create a live broadcast where you could fly around in a virtual drone and observe the scene as if you were using a free moving camera in a video game.

[Lurker]

A faster 3d rendering system for CG software. Like a alternative to Vray.

[krishnamohan]

Seems like self driving problem got solved! Given enough data, someday these algos will be able to understand everything on road. Just like Elon said, cameras would be more than enough for level 5 driving!

[ubiquitis]

Could it possibly be applied to radiology, predicting a 3d image from a 2d radiograph? Could cross-train it with actual CBCT-3d scan data sets, probably MRI as well....

[DaveGamesVT]

Very interesting. This might be the future of 3D rendering in games and movies. Have the system only do a small amount of processing, give it to the AI and have the AI guess at the rest.

[raphirau]

I was really looking forward to this paper! Great summary!

[pc]

The line between Photoshop and a full on 3D Modelling program is going to blur heavily IMO. Soon amateurs will be able to create video games out of Salvador Dali pictures. What a time to be alive!

[kevalan]

Can this also be used in higher dimensions? (Currently this is about 3D imaging, but does it work on an abstract dataset of say 100D?)