How To Send Inputs to Multiple Windows / Minimized Windows with Python. Or Die Trying

Let's explore using SendMessage to send automated inputs to multiple windows at once, or to windows that are minimized or in the background. I'll share my progress and discuss the suitability of using this method for botting.

Today I want to answer the most asked question on my channel:
- How do I send mouse and keyboard inputs to multiple windows at once
- How do I send inputs to a minimized window or background window
- How can I keep using my mouse and keyboard while also sending automated inputs?

I'm guessing most of you want to know how to do this because you've written a bot, and you want to run multiple bots at once, or you want your bot to run in the background while you continue using your computer.

And my initial reaction when I started getting this question was: just use virtual machines. You could have multiple VM's running in the background and those would be isolated from whatever you're doing on your main desktop. I've confirmed that this does work, but the problem is the performance of virtual machines is typically not very good. At least with Hyper-V, even basic games show considerable lag. You might be able to find better performance by setting up hardware graphics acceleration for your VM, but that's not the route I decided to explore today.

So what other options do we have?

About a year ago now, as I was working on my OpenCV series, I got a tip from a viewer telling me about this SendMessage() function he was using to send inputs to a window in the background. It's a Windows API function he was calling via pywin32, and it allows you to send messages, including mouse and keyboard inputs, to a specific window. And there's a couple StackOverflow discussions about SendMessage(), but other than that, there really isn't a whole lot I could find about working with this. So let's try it out and see what we can do with it.

I first tested a simple example with Notepad, and this gave me more problems than I was expecting.

The Notepad window has an inner window called 'Edit' that you need to target if you want to send inputs to the main text area. Even after I solved that, I wasn't able to get any results using WM_KEYDOWN and WM_KEYUP. Only by using WM_CHAR with ord() was I able to get text to appear in Notepad. Both SendMessage() and PostMessage() work this way.

Next I decided to take this to a simple browser game. I found that WM_KEYDOWN and WM_KEYUP did work in Chrome, and there weren't any inner windows to worry about either.

The first challenge I faced here was how to press down multiple keys at the same time. If you sleep() between WM_KEYDOWN and WM_KEYUP that's blocking, so other code can't execute at the same time. You could solve this by using threading, but this time I decided to use sched to schedule each command in queue, and then run that queue once it's been built. This worked really well for one game, so now I wanted to scale it up to play games in multiple windows.

Unfortunately in Chrome, I couldn't get SendMessage to work when the window wasn't focused. So I ended up using SetForegroundWindow() to quickly bring focus to the window that I'm sending the next command to. This does work, but because setting the foreground window takes a little bit of time you have to be careful to not have two commands run at the same time. I wasn't very careful about how I avoided those collisions here, so that's something you'll want to develop if you plan on using this method of quickly swapping between windows. The more copies of the game you're running, the more likely you are to run into these conflicts.

As a side note, if you're going to be swapping window focus anyway, you might find better performance using SendInput(), or even just PyAutoGUI or PyDirectInput as you normally would. The only benefit that SendMessage() or PostMessage() give us is that we can target a specific window handle with them.

So now I kinda have a solution for botting in multiple windows at once, but we still have the issue of wanting to send inputs to minimized or unfocused windows so that we can run our bots in the background.

I found that how SendMessage and PostMessage behave are very dependent on what program you're trying to automate. For example with Firefox, I was able to send inputs to Mario even when the browser was minimized or in the background. But this wasn't quite a perfect solution because Firefox was laggy compared to Chrome, and I couldn't get SendMessage to work with multiple tabs or multiple windows in Firefox unless I resorted back to our SetForegroundWindow method.

#programming #python

[NobungaGames]

Please Don’t stop uploading!!! You have probably the most helpful AI videos I’ve found so far! Please keep going

[marceloviana]

hey bro, please dont quit, the most pleasure content of yours is how to make bots, so you should focus on that. Your audience will increase easily, focus on various types of games.

[vitordomingos]

felt actually that we were " Learning" and exploring together XD amazing man,

[aarongill]

Subscribed when you said "I don't know lets figure it out." Thanks for these videos

[jinnie]

I found your channel just recently and gotta say your little series on writing a bot got me back into coding and I'm having a lot of fun! :)
So thank you kind sir

[compinerd]

the way you teach and show how you go about solving a problem and give us tools at hand and show us how you figure out stuff is exceptional. i thank you for all the work you've put in your videos so far.

[umapessoa]

Hi, you don't upload videos in a while, hope you're fine, your videos are pretty helpful, please if you're reading this, we need more content like yours, make more videos when you can.

[realhippohenry]

Don’t know if you stopped making videos but, just found your channel and learnt so much useful stuff that I’ve always wanted to figure out! Thanks for the great info and I hope to see new videos from you in the future.

[kentalontaga]

Thank you!! ive been waiting for your uploads

[keklmao]

I just wanted to come to your newest video to let you know i love the way you teach, and have been learning alot! thank you!

[ByteBeacon]

I was trying to do this for a browser game that I was automating. In the middle of trying this I found a solution specific to browsers. You can use playwright or selenium to automate a browser. This includes sending mouse and keyboard inputs on specific elements and/or coordinates of canvases. These libraries include a screenshot function which you can pair up with opencv and do image recognition. This can all work in the background. The browser doesn't have to be maximized.

[SaituV]

Subscribed to you some time ago. Never regretted it. Hope you are well.

[CommentFrom]

You got any plans for more videos? Really loved everything you've uploaded so far

[張修瑋-ds]

Hey, just want to say thank you for making these videos! They are so much helpful :)

[ktc]

Bro your content is the best python content I have ever had the pleasure to watch. I have learned so much from you in like 4 videos. I know you haven't uploaded in a while but I hope you're doing well. All the best my man, cheers!

[Linda-mjcz]

Love the new hairstyle :$
Great video!

[lotto]

This is way too specific but its exactly what I'm looking for... Thanks for uploading

[w]

Thanks a lot for all your videos. You have helped me a lot with deeper understanding of some of the subjects. THANKS A LOT!

[espvolt]

abandoned looking for something like this for months this is sick

[gm]

I think that writeprocessmemory might just be the solution.
In theory there's a 1byte or 4byte value that determines if the window is active or not. This value is then used to allow/disallow inputs, if the value is flipped to a 1 then it will start reading keyboard inputs.

I worked a lot with assembly and many applications prevent custom inputs, but they do still allow modified Xinput inputs, and that can also be an alternative.