This AI Makes Digital Copies of Humans! 👤

📝 The paper "The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting" is available here:

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bryan Learn, Christian Ahlin, Eric Haddad, Eric Martel, Gordon Child, Ivo Galic, Jace O'Brien, Javier Bustamante, John Le, Jonas, Kenneth Davis, Klaus Busse, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Mark Oates, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.

Károly Zsolnai-Fehér's links:

#vr

[juliandarley]

brilliant! it surely cannot be long now before we can create feature film scenes and perhaps avoid the expensive capture set-up by using eg Reallusion's Character Creator system. the relighting and shadow-creation aspect shown in this video and from the 2019 paper is simply fantastic. trying to match the lighting between CG scene and green-screen subject is really hellishly difficult, and then what happens if, during the edit, you decide the mood is wrong and you want to change the lighting in a major way? can't wait to see what is going to happen next!

[otheraccount]

The first and second humans didnt look real to me, the juggling one seems quite convincing.

[Zeekar]

Super impressive. It looked immediately uncanny, but probably because I watch these enough to know what to look for... For example, hair looks a bit plastic and clothing wrinkles seem stiff in motion. I wouldn't be shocked if those weaknesses have already been fixed since this is from 2019...

[MemesnShet]

Wow i don't know what impressed me more: The realism of the model or how realistically it integrates it with the environment and gets affected by it.This could eventually be like a body deepfake:Actors might sell the rights for their models so their families can get paid royalties from new movies long after they died or even when they are alive and don't feel like actually acting

[_the_base]

What a time to be able to enter the matrix!

[IceMetalPunk]

Lighting matching has always been, to me, the main issue with making realistic AR composites. No matter how well-modeled, well-textured, well-animated, and well-tracked the virtual object is, when the lighting doesn't match the environment, it always feels like a cut-and-paste image. This kind of light-remapping AI is exactly what AR compositing needs!

[kawag]

Similar capture tech has been used in Free Guy albeit with a smaller footprint and RED cameras. That was just for the face capture and I felt it was dipping only occasionally to the uncanny valley. Regardless the date that this paper was published impressed me more!

[Indecirphrus]

Packaging an old paper as an exciting twist is both legit and brilliant.

[cheydinal]

I literally thought that up just a few months ago (not 100% sure if I subconsciously got this from a paper like this though): Measure how something reflects light by shining light of different colors and brightness etc on it in quick succession, and with that you can then make a very accurate model of that thing

[fugslayernominee]

Amazing!
I yearn the day when i'll watch you saying, “just two more paper down the line before we can upload our consciousness and live forever"

[johnclark]

Dang, if they’ve already made digital human copies 2 years ago, 2 years from now they’re going to replace us with those copies!

[yosefpeter]

I love how Black Panther narrated this, as Wakandan share their knowledge to the open world.

[MrSpartaa]

This one blew my mind. Imagine stuff like that being used in games.

[panictech]

I just love it when this guy says "WHAT A TIME TO BE ALIVE" makes me feel hopeful and im pessimistic person

[GGCannon]

@Two Minute Papers
This added together with Sebastian Starke motion papers and the AI Stuntman by the 4 Lees of Seoul National University will be amazing!
Just imagine being able to digitalize yourself or anyone and animating them with AI generated motion that can be recombined and that is adaptative, physics accurate and quick to render in real time.

[liontuga]

This tech is mind-blowing. This video immediately reminded me of a VR music video from Billy Corgan - Aeronaut. It's not only a beautiful song from Billy, but you also literally see him play the piano in front of you. What a treat!
(It's free on Steam.)

[gargaduk]

To me, only the old guy juggling was convincing. Well, almost convincing, apart from the jittery animation.

[marthinus.x]

I can see this feathering in very nicely with VR and could very likely be used to create true holo-movies with digital playback volumes - like some form of VR assisted Holodeck! 🤓 So exciting!

[thelawgameplaywithcommenta]

The first two were obvious. but that third guy had me scratching my head. It looked so real!!!

[fietae]

That Dome is the first useful usage of rbg lighitng i have ever seen lol