Web Scraping for Beginners

Learn how to get started with retrieving content automatically from the web with these techniques and tools of the trade. Note that this video is not meant to be a complete guide to web scraping but an aid to get started.

[gregorymagarshak]

It says that it is released under MIT License but there is no LICENSE file in the repo… and the files say Copyright, All Rights Reserved. So how can we be using it in our projects? Do we need special permission?

[cubiclesoft]

In my channel alerts, I see: "can you do a video to explain how to do this, this is very vague. I will look at Ultimate web scraper tutorial, is there an easier way to find the request, selenium and cherrio and others i dont th..." (the message cuts off). However, I don't see any actual comment on this video anywhere. Either the comment was deleted later or was blocked by YouTube for some reason. I'll admit that this video is a bit vague. Each situation in web scraping is unique but there are general patterns to follow. You eventually learn what to look for and certain types of requests start to stick out more than others as being important. I always start in the web browser so I know what it is I'm looking for when I go to write code because sifting through thousands of lines of HTML and XHR response data from a command-line is pretty difficult. The browser allows me to laser-focus my efforts on what network requests to make instead of trying to load every single thing like a real browser does. If you are just starting out, web scraping someone else's content can be kind of daunting. I recommend starting into web scraping by scraping your own website(s) that you have built. Since you control your own website, you won't irritate a sysadmin and you already know how your own site works. It's a good way to get your feet wet before taking on a more significant challenge.

[rensvanmelis]

Could you maybe go more in depth about the asynchronous part of web scraping?

[SachinKumarSinha]

How to handle consent in your library?