Real Time AI HAND POSE Estimation with Javascript, Tensorflow.JS and React.JS

Want to get started with gesture detection?

Or maybe you're interested in sign language recognition?

Well...hand pose recognition can help you along your way. Hand pose recognition uses deep learning to detect different landmarks in your hand. By joining these landmarks you're able build a 3D mesh of your hand, and specifically the pose it's in. In this video you'll learn how to build your very own hand pose estimation app with Javascript, Tensorflow.JS and React.JS.

In this video you'll learn how to:
1. Setup a pre-trained hand pose deep learning model
2. Build a hand pose app using your webcam
3. Detect hand landmarks in real time

References Mentioned:

Oh, and don't forget to connect with me!

Happy coding!
Nick

P.s. Let me know how you go and drop a comment if you need a hand!

[akshitdayal]

I'm so glad I found your channel out here! Everything you post is extremely useful and beginner friendly!!

[woutvandesompele]

THIS is what you call a flawless tutorial!! Thank you so much!

[igor_cojocaru]

Thank you!
Two days trying to set up this in react and finally found your video!
Cool!
Subscribed

[diegocaumont]

Love to follow along with ur tutorials man, clear, concise and informative. Great stuff.

[ifvictorthen]

Great video!! For those who will run this code: the load model shouldn't be done every frame and the webcam properties can be saved in some local state (hook) for performance.

[hiteshsharma]

Hi Nicholas, the drawn hand and actual hand in video were not coming aligned for me. I figured that's because teh width and height of the canvas and webcam element were hardcoded to 640x480. Changing them to videoWidth / videoHeight did the trick.
Works great !

[영상일기-hh]

Thank you for the tutorial! this is really cool. I just have one question. When I do it until 25:00 where I can see the landmark points appear on top of my hand, the landmark somehow appears bigger than my hand(whereas your landmark is the same size as your hands). Do you know how I could fix this problem? Thanks in advance!

[きさま-xv]

was a greatly nice stuff
you saved my time

[ashishhameed]

hey nicholas whan I cloned the repo and ran it in my browser it was very slow. Do you have any idea on how to optimize it?

[Dara-gqjl]

Very nice video learned alot but I had one question how do i make ctx draw both hands? thanks

[nobodyeverybody]

Thank you for your tutorial, I could go along with it, however, it is just a prototype and not an actual app. So, you may talk more about how to make it responsive to the size of the browser and device and how to make it run faster without any performance problems. tanx!

[kenmiles]

Thanks for these tutorials! I have just one doubt, suppose I wanted to do these predictions only when we press a button, i.e. the webcam and predictions process starts and stop when we press a button. Can you suggest how to implement this?

[midor_bare]

could I use real-time hand, body, and face recognition all at the same time for creating a real-time moving 3d produced avatar?

[AMRUTHAK-qusl]

Thank you for the wonderful explanation. Could you please give a tutorial on hand gesture recognition using LSTM model

[wgalloPT]

QUESTION: I know its been a while youve done this, but my question is what if I want to add the forearm segment to the hand for the wrist joint? any tutorial that youve done that could help me with that? Thank you!!

[saeidrezatalaeikhozani]

Hello, Thanks for your great video. I have a question, when I try to use the app on the cell phone browser, it couldn't detect the the points on the hand correctly. The points are not attached to the hand and totally not working well on the browsers of mobile phones. Can you let me know how I can fix it please. Thanks again :)

[owenizedd]

For experienced developer, real things start at 5:50

[kalyanchakrig]

Thank you so much . The way you explain is simply super. Can we do the same for wrist to do try on for watches.... Thanks again.

[petitblou]

At the stage to run to train the model, it shows me:

MacBook-Pro RealTimeObjectDetection % python --num_train_steps=10000
Traceback (most recent call last):
File "Tensorflow/models/research/object_detection/model_main_tf2.py", line 30, in <module>
from absl import flags

ImportError: No module named absl


While I'm stilll having the absl package installed
Could you please help

[riturajsingh]

Hi Nicholas. I understood all of the video and wrote the code following you but i didnt understand only one thing. i would be really nice if you explain why you did this

// Set video height and width

// but why is this important