Scene Responsiveness for Visuotactile Illusions in Mixed Reality (Full Video)

Scene Responsiveness deepens the illusion of passthrough AR so that virtual actions and virtual characters seemingly affect the physical scene. We toggle the reality state of objects from physical to virtualized by building on situated digital twins and a novel graphics and shader pipeline. This allows the objects to be manipulated in all ways virtuality affords while maintaining visual coherence and visuotactile consistency.

Mohamed Kari, Reinhard Schütte, and Raj Sodhi. Scene Responsiveness for Visuotactile Illusions in Mixed Reality. In: ACM Symposium on User Interface Software Technology (UIST '23).

Research conducted at Meta Reality Labs Research

#mixedreality #augmentedreality #metaverse #mr #xr #ar #vr #spatialcomputing

[theadamantitewaffle]

the naked minion streaking through at 3:44 is the best part

[undisco]

Yeah, this is genuinely some next level stuff. In terms of game graphic / interaction capabilities and MR merging/blending. Have not seen the term 'visuotactile' before. Nice work.

[AeroD]

Incredible! And with Gaussian Splats, the illusion will only get even better!! I love this research! Such astonishing work!

[ChristopherCricketWallace]

The safety challenges around this are staggering

[LiVeWiRez]

🤯🤯🤯🤯 mind blown. This would make for some real trippy games, literally. 🤔 Maybe even dangerous if the objects don't re-appear fast enough while moving towards it.

[chanceslaughter]

This is so freaking cool this is exactly what I want to do for work is develop this technology

[guidewired]

This is great! Excited to experience something like this some day

[TicTac-gm]

In the real-world descriptions of vehicle accident investigation, a common "defense" of collision is "Officer, I did not see them/it"
Usually this means the object was in view but the drivers ATTENTION WAS FOCUSED ELSEWHERE.
That's the key. And a major problem.
The developers of this attempt to address this safety concern by removing the mask (making the real object that was invisible, visible again if one gets too close-- in efforts to avoid collision)
But it doesn't always work that way.
In real life, humans visual attention can be focused on something else even if the danger object reappears in their field of view.
The same with our other senses.
This happens all the time in real life, and WILL happen with this technology--- even if an automatic"alarm noise" is introduced. Think "Officer, I did not see them and I did not hear a horn"
My DJI drone, for example, has proximity detection and both visual and audio warnings.
I can't tell you how many times I failed to even notice they were on--- my ATTENTION WAS FOCUSED ELSEWHERE.
Be careful. This tech could most definitely result in unintended injuries.

[MetaPrintD]

Wow this cool! I hadn't really thought of this before but it makes sense to do for certain applications. I liked the elevator one...would be good to have an AI stable diffusion built in that could help blend and augment the 2 together...

[TheMirrorslash]

Finally some amazing AR AI integrations. Well done!!

[Engmatic]

This is awesome and mind bending indeed. I'm able to follow, but just barely lol...

All that said. Outside of realtime mapping your physicsl space and reprojecting a photorealistic/static version laid over your actual space? What is the actual use case for such application?

[BenGeskinTech]

This is incredible 🤯 OMG future is exciting

[neurolit]

this is amazing! i’ve always wanted to work with vr/ar and neural interfacing. seeing demos like these are really inspiring. i’m curious how objects like the real cart are segmented and modeled. especially since ur still manually superimposing a model of it. the occlusion seems like a very solvable problem tho given how the current build effectively removes irl objects in space.

unity is great :) would love to cont the convo

shoaib

[bradyhem]

This is actually very insane. It looks so good

[VolkerTyranuke]

Aus technischer Sicht, sind es unglaublich großartige Zeiten, in denen wir leben 🤯

[benirodriguez]

If you forget to take the VR headset off and clean your room, you'll be disappointed when you take the headset off 😂

Besides from that... Amazing stuff!!

[mhnoni]

2:44 how is the scanning works here? need an iPhone?

[StormyDoesVR]

In terms of technological advancement, this is crazy stuff. Using AI to make items disappear that are actually in your real surroundings is a terrible idea though, no way the computer is going to be able to keep up all the time and keep you from whacking stuff

[Stonehenge.]

So fun to consider adding robotics as little stage makers :)

[jcl]

I hope you take care and not trip over an actual chair in front of your stairs that was occluded. 😊