Real-time Object Detection - OpenCV Object Detection in Games #5

Learn how to detect objects inside a game window in real-time using OpenCV.

0:37 WindowCapture updates
3:28 Integrating findClickPositions()
5:18 Updating main
7:20 Demonstration of real-time detection working!
7:59 Creating a Vision class
14:12 Browser game demonstration
15:56 Shortcomings of this object detection

In the first three parts of this tutorial series, we learned how to use OpenCV's matchTemplate() to detect objects inside an image. Then in part #4, we learned how to quickly capture screenshots and display them as a video stream inside of an OpenCV window. Now we're ready to combine together everything we've learned so far in order to detect objects in real-time.

I've also changed the constructor to accept None for the window_name. When given no window name, we'll have WindowCapture just capture the entire screen. I've found that certain applications return a black screen (Chrome), or do not update after the initial image capture (Firefox), and capturing the entire screen can be a suitable fallback plan in those cases.

The first parameter in findClickPositions() is the needle image we want to search for. You'll want to manually take a screenshot and crop out the item you want to match (like we did in part #1). For the second parameter we'll pass in the screenshot returned by WindowCapture's get_screenshot(). Then set your threshold, and enable one of the debug modes.

We'll call our new class Vision, and let's read in the needle image in the constructor so that we don't need to repeat that step during every iteration of our main loop. We can also assign our matchTemplate() comparison method here. This will allow us to use different methods for different visions.

We'll want to make findClickPositions() a method inside of the Vision class. To do that, get your tabs right, and remember to add self as the first parameter. We also no longer need the needle_img_path parameter in this function, nor the other steps that are now handled by the construtor. Remember to update all usages of those variables that are now class properties to use self.. I'm also going to take this opportunity to change this function name to simply find().

[johngalt]

Your videos are better than any paid service. This is great!

[Alexander_Meyer]

Looking sharp with the new hair my man!

[scoringdigitsson.]

Your contents are top notch. You deserve more subs!

[dieweltentdecker]

Yeah, finally a new video. I've been waiting for this video for 2 weeks. When are new videos coming?

[kelvinmandlik]

Finally dude i have been waiting for this so long
I have watched the video entirely but don't stop this series please.
Waiting for next one,
by the way, nice video

[jaegyunlee]

Wow this video is worth watching. I can't believe that a lecture with this quality is just uploaded on Youtube for free!

[thezichko]

Hey, I'd like to thank you very much. I've been wanting to learn to code for a while but have felt so small compared to how powerful programming can be. I'm extremely thankful for your videos. I'm now taking what I've been learning from you and other talented people to help me with my robotic/AI ideas I've been dreaming of for a long time now. Who knew it took learning code by playing games would be one of the best ways for me to learn but it helped a ton. Thank you! I still get annoyed with the code still but I take a break and a few hours or the next day I've figured it out.

[thedeveloper]

you're one of my favorite programmers on youtube! thanks for the video and nice hair!

[Ab-cjgl]

man this channel is fooking amazing
extremely underrated

[Гусьэкономит]

It has been two years already, where is the continuation? Why did you abandon your channel? You explain better than all my teachers! You're the best! It has been 2 years already, perhaps you have improved object detection, bot performance, and their movement between points, or maybe there's something else interesting to see!

[Lasnikers]

I love the tutorials and I am just watching and not even coding along (I only program in Java)

[sensbo]

I stopped already coding 10 years ago and doing other things. I started playing a game some month ago which consumes a lot of time and look for opportunities for automate it... and agian think about to start coding with the new tools, like py. Its a loop which will never ends... thank you for sharing that tutorial.

[brvhixu]

I created a farming bot for an mmo that clears a stage of bad guys, and at the end runs around to look for a relic and go to it to leave the mission. I used 3 needle images at different distances, but it is extremely hard for the bot to find since the relic has fire coming out of it and the level is on a fiery magma rock surface. Im really excited for the next video, because this has really been the final frontier for my bot that I can't seem to get through as well as id like. Thank you for the great content.

[SquallHart]

You're an EXCELLENT teacher. Thank you!

[Nish-dy]

You are awesome "Ben", thanks for amazing learning session!

[eduardopetecof]

Very nice series, I'm really enjoying it and learning at lot too. Thanks for putting all the effort into these videos. I'm strugling a bit to adapt the WindowCapture code to my Ubuntu OS, but hopefully I will get there. I can feel Deep Learning coming and I can't wait to see it :)

[PainfulDeath]

This is great stuff, commenting so youtube algorithms show it to more people :)

[fabiorodrigo]

Still getting a black screen. The script certanly finds the window, but its all black. Anyone else with the same problem?

[thanhhaphan]

Cant wait to watch the video. Thanks Ben!

[bowlingballs]

Man, you are amazing, as a beginner I learned a lot.