Robust Camera Orientation Tracking Using Manhattan World Hypothesis

The video shows a robust, purely image-based orientation tracking in a Manhattan world. The cube in the picture shows the orientation of the world coordinate system (Manhattan world). 3D object edges that are parallel to the axes of the Cartesian world coordinate system are shown in red, green and blue.

The method is applied to camera images of the Intel Realsense T265.As clearly visible in the video, it is possible to estimate the camera orientation in a non-ideal Manhattan world: There are people, plants and oblique objects in the scene.

We use the so-called "top-down" approach: a hypothesis for the orientation of a camera in space is tested on the features (lines segments) recognized in the image and not derived from them "bottom-up". By discretizing the the solution space (3D rotation space), rotation hypotheses and their associated vanishing points are calculated offline. In order to determine the best rotation hypothesis, each rotation hypothesis in the neighbourhood of the rotation vector from the previos frame is tested. The best rotation is chosen based on how many lines of the image can be assigned to the associated vanishing points.