Apple visionOS Object Tracking Is Here! BUT DOES It Work Well?

Today, I would like to share what I’ve learned about Apple’s Object Tracking features, recently announced as part of visionOS 2.0 Preview.

This video will walk you through the entire workflow: image capturing/scanning, object reconstruction, machine learning training with spatial object tracking, and integrating trained objects into Xcode.

📚 Vision Pro Object Tracking Chapters:
00:00 - Apple visionOS Object Tracking Introduction & Questions
00:29 - visionOS Object Tracking Requirements
01:17 - Preparing Physical Objects For Object Reconstruction
02:12 - Generating USDZ Files (3D Models) With Reality Composer Pro
03:10 - Training Models With Create ML Object Tracking Features
08:27 - Integrating Trained Object Tracking Models With Xcode
19:05 - Running Object Tracking Demo With The Apple Vision Pro
20:27 - Adding Reference Object USDZ Model For Additional Visualizations
21:02 - Object Tracking Demo With Reference Object Model Rendering
21:57 - Outro (Final Thoughts)

📌 Helpful Object Tracking Resources:

🥽 Learn & Get my XR Courses from:

👉 My Blog / 🔥 Newsletter (Subscribe to get up to date XR news)

#AppleVisionPro #VisionPro #Apple

[dilmerv]

I also recommend going over all the resources available from the video description.

[vany]

This is a very cool idea, it was interesting to make something like the face masks that we applied. In real time.

[bencrystalofficial]

This is incredible! Can't wait for 2.0 to drop more formally :)

[renatos.]

Thanks for these detailed instructions! 😀

[yifansun]

Cool Video! I just tried to use a CAD model to train the reference object. However, it didn’t work. I am curious about how to config the usdz material for a CAD model if we do not want to use the scanned version which is not as accurate as the CAD😅😅

[xygu-uo]

Very cool feature. I have noticed that there is another newer session called SpatialTrackingSession which can provide tracked object's world position, can the one you used also output some similar data?

[kentarotakahira]

Thank you, Dilmer. It's very easy to follow. Is it possible to combine this object-tracking feature with the Unity AR project? (Like Ar foundation etc.)

[juliancastiblanco]

Great video, you never disappoint. I have a question tho. How do you access Guided Capture iOS on your phone? The link in description is unavailable

[justinchang]

Does the tracking generalize well or is it specific to the physical object you scanned?

Specifically I'm wondering if you can scan 1 car (compact sedan) but have the app work for all cars (compact sedans)?

[GuilhermeGouveia-nxrp]

Hi Dilmer
First of all, thanks for your content !

Do you think that The Object Tracking will be accessible from Unity side similar to Vuforia model targets

[paulalexandrupop]

What's the limit on the number of reference objects that you can track?
Couldn't find that anywhere in the docs.

[bencrystalofficial]

Do you happen to know if I update to the Beta if there are errors, is it possible to downgrade by factory resetting the device? Or will it factory reset back to the same beta?

[erikmeilner]

@dilmerv thank you for your great job. I have a problem. Could you please help me? I want to attach a reality view with a usdz & an attachment(SwiftUI view) to the object tracking anchor how to do this?

[antonimalachowski]

It seems like it should be possible to reduce tracking latency on stationary objects if you integrate the tracking box with input from the gyros and accelerometers. Just a thought. Anyways, thanks for the upload!

[kkkaran]

Can we use this as native library and plug it with Unity? to send the bounding box pose?

[anirudhnv]

What if I already have the 3D model of the object I want to track as an fbx or cad model. Can I use that to track the real world object?

[NikolajStausbl]

Have you tested if this works and tracks as well in an iOS app? Or is it VisionPro only?

[antdx]

Does it way more battery doing that as it constantly looks for the objects and needs to know their positions?

I assume if you have to make it search for more than 2, it will drop the FPS heavily and be unusable then the app crashes?

[Polerface]

Is this anything even close to this in the Meta Quest 3 universe? I can't find a way to track real life objects. Do you know any solutions? Even with external hardware..

[CodyMaverickSurfsUp]

what's the lag? How about accuracy?