Free Motion Tracking plugin for TouchDesigner: Face, Hand, Pose Tracking with MediaPipe Plugin

Hey! In this tutorial, we'll go over the new free open-sourced MediaPipe plugin for TouchDesigner that's GPU accelerated and works on Mac and PC with no instillation. The plugin supports face, hand, pose and object tracking with multi-person face detection, object detection, and image segmentation / background removal.

0:00 Intro and Overview
1:44 Project Setup
5:43 Face Landmark Detection
9:37 Hand Tracking + Gesture Recognition
13:49 Pose Tracking
16:37 Object Tracking
18:05 Multi Person Face Detection
19:21 Image Classification
20:56 Image Segmentation / Background Removal
23:58 On Mac Pipe Video into MediaPipe with OBS
26:26 On PC Pipe Video into MediaPipe with SpoutCam
28:30 Outro

[arthoefmann]

Can I just say this is brilliant work, thank you so much for making it accessible, as an artist and designer working and exploring touchgdesigner tools like these make it so much easier and more fun to learn and experiment!!

[feelzlab]

wow! I just get to know about MediaPipe - thank you so much for creating this and making this tools more accessible for individuals like me who have visual ideas but are not as technical as one can be.

[nightshining]

Thanks for the plugin! For anyone having issues with the MediaPipe tox loading forever, the last version of 2022 (.35320) works for me on an M1 mac. I just avoided having to upgrade my commercial license for another $300 over this.

[AliTanUcer]

this is phenomenal!!... i was about the tackle the same thing and wishing to find time to port everything to GPU.. now we have it.. amazing.

[gregderivative]

o my, brilliant tracking without special hardware. great explanations.

[devicenotfound]

Thanks for sharing such a helpful video! I never thought TouchDesigner could do this, and it's truly inspiring. Your video also has shown exactly what I've been looking for - how to superimpose TouchDesigner output to OBS.. Your effort is greatly appreciated!

[pagarreton]

Wow this is incredible! I am really happy it also works for Mac computers!

[OzanTezvaran]

This looks amazing, cannot wait to check it!

[yoli]

i would be really grateful to understand the spoutcam for windows in depth for the mediapipeline plugin. Thank you

[jimkuehn]

Sick sick sick. Am rolling into a project that might benefit from your great work. Perfect timing

[christopheluez]

Incredible content ! Verry smooth, everything is working perfectly ! (Macbookpro 2021 M1pro, lastest TD version)

[borjonx]

When looking for channel names(like around @20:37 & earlier) you can also just add a SELECT and click the black triangle next to "Channel Name" for a full list.

[markus_knoedel]

WOW, I will be off grid now for some time to figure all of this out. Thank you. So helpful.

[cele_stin]

This is unbelievable, thank you guys!

[BossCODINGplease]

This is truely amazing, thanks for all the hardwork

[eldelbaar]

Thanks a lot for making this project public! it helps a lot for people that is starting

[gianlucacolia]

This plugin is amazing, I hope you keep updating it. thank you so much for sharing it!!!

[jagvalverde]

According to the workflow logic of my project, I would need the image input to be a pre-recorded video, but without using OBS, only using TouchDesigner. Would that be possible?

I want to create different automations with different videos, and I don’t think OBS will allow me that configuration.

I have no words to express my gratitude for your contribution. It’s incredible. Thank you so much.

[knast]

Hey guys, first of all: huge thank you for sharing and working on this - This is soo sick! Everything is working as intended for me (2023 version of TD) But I want to overlay the image segmentation with my original webcam-in and I'm having issues with it just showing a 'black screen'. Even if I try with the SpoutCam I get a black screen or saying 'No Active Sender Found'. I hear that you have to use TD 32bit version for SpoutCam to work, but is that true?

Maybe there's a way to get the camera and the image segmentation out on the same time to composit or overlay the two?

All the best.

[mustaTraceur]

Hi, I hope you can help me.
I just download the mediapipe plugin, so I open the main project MediaPipe TouchDesigner.toe then I Select my main camera and it doesn't show any image, but when i use a videodevice_in it automatically use my main camera.
Therefore almost every subsequent node show me some kind of error
Im using a macbook air M1, so, the main camera is a facetime camera.
Also, when I drag and drop some of the .tox at an empty Touchdesigner project it just doesnt drop at the workspace. Any idea if Im the one thats doing something wrong or someting else happening?