How to Make 2500 HTTP Requests in 2 Seconds with Async & Await

This is a comparison about how to use Async and Asynio with AIOHttp and Python vs using threads and concurrent futures to best understand how we could make several thousand http requests in just a few seconds. Learning how to do this and understanding how it works will help you when it comes to running your own servers and web services, and stress testing any API environments you offer.

Support Me:

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

[jmoz]

You do not need to explicitly create task on line 18

[victorhaynes]

I can't believe how fast you got to the point. thank you for reading the room

[artemfagradyan]

Thanks for your video!! Tried to send multiple async request for some api but my previous code was unefficient and your example helped me!!

[fmanca]

John, your channel is absolutely fantastic! congratulations!

[eduardocasanova-personal]

Thanks for the video, makes me appreciate go routines and their simplicity even more :)

[swordlion]

ThreadpoolExecutor is slower than Thread but saves memory. You don't need to use asyncio for waiting. You can use joins, barriers, wait groups and mutex locks with Thread to achieve the same. It comes down to preference, although asyncio is more streamlined, and thread is better suited for manual optimization that requires utmost speed and care.

[BringMe_Back]

Awesome man, I was working with these things Today, I'll try this one too ♥️♥️♥️🙏

[ihateorangecat]

hey sir👋
as a self taught dev You are one of my inspiration 🙌🙌🙌
i started learning web scraping recently. i downloaded some videos about scrapy and BeautifulSoup tutorials of yours and
i followed along them and found pretty comprehensive and clear i did learn better by your videos.
i do hope more and more tutorial videos!!!
thanks million times !🙏

[hristijansaveski]

Thank you good sir, you are a master at this!! You have helped me land and keep my data scraping job!! Thank you so much, truly an inspiration :))

[ErikS-]

"how to build a ddos attacker..."

[terrascape]

hi john, or anyone who knows, I keep getting the error "UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 28855-28856: invalid continuation byte", do you have a work around for this? the ones that I found on google I can't seem to implement correctly.. much thanks!

[ericxls]

Very good vid!! I just finished making use of concurrent futures (based on your previous vid) and it speed up my code considerably! Looks like I have the potential to speed up further 😀. Will making a lot of requests at the same time slow down the source server, thus passing the waiting time to the server?

[mrjt]

hey thanks, good info.

how to run 'subprocess' Asynchronously, while I dont want to use 'requests.content' to download

subprocess.run(["yt-dlp", download_link, "-o", f"{output_dir}/{episode_name}.%(ext)s"], shell=True, stdout=PIPE)

Also, What would you prefer Multi-Threading or Async in this case ?

[acatisfinetoo]

I was looking for a good explaination of the difference between async and multithreading...so threading is for doing things in parellel and async is for waiting for future tasks to complete but dosn't stall the current program?

[saranpun]

While using executors, if the capacity of tomcat to process is set to 200 only. What happens after 200 request? Will we have to wait till 200 request are processed by the server? Or, it will pick up some request when call passes from server to db.

[vincentdigiusto]

very instructive thank you John, I think that scrapy uses async requests, that's why some scraping jobs can be impressive quick with scrapy

[androidmod]

Nicely done, a Proxy or socks5 "which i believe works with requests" will do the trick bypassing the traffic limit. But how can i implement it in this scenario? Thanks John.

[aabmets]

Thanks, I'm gonna implement this in my "yfrake" package (PyPI).

[bobong]

Hi good sir thanks for the vid and most importantly, being actively engaging with people in the comment section Cheers

[redaoutarid]

Thanks for your helpful videos.
Please, have you any idea how to avoid dat*ad*ome protection ?