Windy City DevFest 2019 - Fusing AI with AR: Custom Markers

Speaker: Stephen Wylie

One very popular area of Machine Learning is object detection, but why stop at just emitting a message that some object has been detected? Let’s take things a step further and use the positional information coming from the machine learning model to draw something in augmented reality over, around, or near the detected object.

This is commonly referred to as marker-based AR, but usually markers are a QR code or other distinctive shape with lots of characteristic features. With Vuforia, you can make custom markers easily, but then you’re locked down into their pricing model. Here, with Tensorflow Lite running on an Android device with ARCore, we can use MobileNet-optimized object detection algorithms to spot whatever you want.

Find out in this talk how you can leverage the basic MobileNet object detection tool in an Android app using Tensorflow Lite to discover where an object of interest is, and then use ARCore to render something around it in 3D space. Then, I will touch on how to train a model for your own needs of classifications, plus some of the pitfalls of trying to render an AR asset relative to arbitrary objects in space.