3D Human Pose Estimation via Intuitive Physics (CVPR 2023)

Estimating 3D humans from images often produces implausible bodies that lean, float, or penetrate the floor. Such methods ignore the fact that bodies are typically supported by the scene. A physics engine can be used to enforce physical plausibility, but these are not differentiable, rely on unrealistic proxy bodies, and are difficult to integrate into existing optimization and learning frameworks. In contrast, we exploit novel intuitive-physics (IP) terms that can be inferred from a 3D SMPL body interacting with the scene. Inspired by biomechanics, we infer the pressure heatmap on the body, the Center of Pressure (CoP) from the heatmap, and the SMPL body’s Center of Mass (CoM). With these, we develop IPMAN, to estimate a 3D body from a color image in a “stable” configuration by encouraging plausible floor contact and overlapping CoP and CoM. Our IP terms are intuitive, easy to implement, fast to compute, differentiable, and can be integrated into existing optimization and regression methods. We evaluate IPMAN on standard datasets and MoYo, a new dataset with synchronized multi-view images, ground-truth 3D bodies with complex poses, body-floor contact, CoM and pressure. IPMAN produces more plausible results than the state of the art, improving accuracy for static poses, while not hurting dynamic ones.

Bibtex:
@inproceedings{tripathi2023ipman,
title = {{3D} Human Pose Estimation via Intuitive Physics},
author = {Tripathi, Shashank and M{\"u}ller, Lea and Huang, Chun-Hao P. and Taheri Omid and Black, Michael J. and Tzionas, Dimitrios},
booktitle = {Conference on Computer Vision and Pattern Recognition ({CVPR})},
year = {2023},
}

[mokhtarzadehhossein]

Great job on understanding the importance of maintaining a stable body and its connection to slow movements like Yoga. I'm curious if you've explored its effectiveness in higher-intensity activities such as fast walking, running, or jumping. Considering deformable surfaces adds another layer to consider, especially when the ground is not always a non-deformed surface, like when people play on grass, for example. It would be an excellent addition to your original concept and research model. Keep up the fantastic work!

[owenlarson]

I’m convinced that this is how we get there! Of course your team is the one to do it.

[WhiteDragon]

I too wonder how it would perform on poses with rapid acceleration where the center of mass is not necessarily supported by a center of pressure directly below. For example, an athlete making a sharp hairpin turn would have significant lean, but because of the acceleration AND gravity, the center of mass would be supported despite the lean.