openDog V2 #8 Hand Gesture Recognition

I thought it would be nice if openDog could respond to hand gesture signals instead of driving with the remote control. I really wanted to try to train a deep learning model to do this also so I can see how well it can distinguish between hand gestures. I used an NVIDIA Jetson Nano and used transfer learning with the tools provided by NVIDIA to retrain an existing vision detection model. For now I'm using digital pins to convey the five signals from the python code to the Teensy which controls the robot, eventually I'll wrap the machine vision script in a ROS wrapper to control the robot.

You can support me on Patreon or buy my Merchandise:
***************************
***************************

Affiliate links - I will get some money of you use them to sign up or buy something:
***************************
***************************

Other socials:
***************************
***************************

Huge thanks to my Patrons, without whom my standard of living would drastically decline. Like, inside out-Farm Foods bag decline. Plus a very special shoutout to Lulzbot, Inc who keep me in LulzBot 3D printers and support me via Patreon.

HARDWARE/SOFTWARE

Below you can also find a lot of the typical tools, equipment and supplies used in my projects:

XROBOTS

Former toy designer, current YouTube maker and general robotics, electrical and mechanical engineer, I’m a fan of doing it yourself and innovation by trial and error. My channel is where I share some of my useful and not-so-useful inventions, designs and maker advice. Iron Man is my go-to cosplay, and 3D printing can solve most issues - broken bolts, missing parts, world hunger, you name it.

XRobots is the community around my content where you can get in touch, share tips and advice, and more build FAQs, schematics and designs are also available.

[Logalactic]

This is the COOLEST thing you've ever built. I love it

[mikeselectricstuff]

Instead of aluminium plates for structure, consider using PCB - weight, strength and heat reistance are comparable to ali, and you can get it drilled and machined extremely cheaply. if you need some heatsinking on, leave the copper on

[WeWato]

James, one common reason for the motor to get hot is if the encoder and motor are no longer properly alligned (someone slipped). Check if your encoder.offset changes after normal/heavy use of the axis and recalibrating.

[tps]

this is under rated. he needs more recognition

[Skyentific]

Nice video! Great progress! For robot dog positioning maybe you can just use tracking camera, for example realsense T265. It works fine with the jetson nano.

[Shreyam_io]

dogo is super excited after getting eyes and brain.., IT JUMPS in JOy.... movements are so natural....Loved it... even though he is a non-living robot..

[creativecityis]

I will miss these videos when you become famous. looking forward to seeing your progress.

[mikeselectricstuff]

instead of screws+threadlock, plastic rivets can be a useful alternative

[matterwiz]

I've just realized that I've been watching you since the Iron Man Hulkbuster Project and it feels like that was 10 years ago... Turns out its only 6 years but that makes it like 30% of my life... I guess what I want to say is thanks for making good robitics videos for such a long time.

[GeeTheBuilder]

Absolutely AMAZING video, James!!! 👏👏

Bring together all the recent gesture stuff to OpenDog is just bloody awesome!!

Well done mate 👍

[D-S-]

Completely unwatchable, every one knows that robot POV shots need to be low resolution and covered in scan lines. It didn't ever make, "bibidi bibidi bibidi" noises or have extra text scrolling on the side.

(Another awesome video, thanks).

[SpencerPaire]

Now *this* is how you do image recognition on the edge! This is such a massive step forward for accessibility technology; now someone who doesn't have training for the controller can use universal gestures to guide the robot around.

Can you implement a point-feature? Some way for it to recognize where your pointing and then use the pathfinding you've developed for Really Useful Robot to go to where you point?

So excited to see this continue to develop!

[morgansmith]

Hey James!
If you want to speed up the process of training a model, rather than taking photos for training you could either:
A: Crowdsource data from the community, I'm sure many people would be happy to help, and this IS an open source project after all
or alternatively.
B: create a model of of the subject (in this case a hand), randomize some values (e.g.: shape, colour, place, and position relative to the view) then take some photos AND train the network on that
ALSO: for some resources id recommend checking out some of Jabil's videos. He does a lot of computer learning and has some pretty ingenious solutions!
looking forward to to making this myself and maybe helping out a bit!
All the best!

[nickgenericusername]

So cool to see how responsive the image recognition software is!

[LokiLeDev]

This is so awesome! I've always loved gesture control it is impressive how easy it is thanks to transfer learning. I am definitely going to do that on my robot.
By the way it is totally OK to use the gpios like you did to send the orders to the teensy. Using ros is not mandatory in every layer of the system, as always keep it simple. For the odometry as others said you can use additional sensors and perform all computation/estimation on the Jetson in a custom ros node, for now you can leave the teensy for controlling the legs in open loop.
Keep up the good work :)

[TheGmodUser]

Using odomotry isnt necessary in ros, it is if your using gmapping, but changing it out for something like hector_slam works great.

Hector dosent need odomotry data, but it can use it, also hector_slam allows for the sensors to tilt, where gmapping dosent like that.

[SunePedersen_dk]

Am I the only one hoping that James one day will reveal that a big team is behind all of his great work? I mean, im pretty well founded in many topics too and really productive compared to most people, but comparing to James, I am not getting anything done....

[miroslavcepl]

Nice work! For the motor heating, make sure all bearings are in one line. Since the motor already has two bearings, adding third can add unwanted forces. Basically you lock straight shaft at three points (which may bend it ever so slightly) and then you force it to rotate, effectively bending it each rotation.

[camdenadams]

I LOVE open dog, keep up the great work!

[r.iyushofficial]

Why is this channel so under rated...???