Don't Start Web Scraping without Doing These First

I've put together a short video with 5 tips that I feel could help you some of you that are new to web scraping with python. I hope they benefit you in some way!

Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases

# Timestamps
00:00 Intro
00:24 Investigate the site
01:40 Practice Parsing Locally
02:32 Write a Plan
04:17 Don't Over complicate it
05:12 Pick the right tool

[shoebshaikh]

By far the best channel on YouTube for web scraping ❤️

[_domdge_]

been following you for a week and i learn so many tips. Thank you!

[khaliqsalawou]

Thank you, John, the tips were really helpful. and I would love it if you can share more of this in the future.

[balazseduard]

You are the best man. Much respect, keep up the good work, I learn a ton from you as a beginner

[DIY-Investors]

John, that was a really helpful (top down) overview which I found very helpful. As a visual learner, I almost need a decision tree diagram to take me down the most appropriate route... thereby taking me to the right set of tools/ routines to use. It’s also helpful to have a video in the 7- 10 minute time range, to focus in on the particular topic in hand. 10 out of 10 from me! 👍

[rtxmax]

Your channel is too good for us scrapers!!!

[theinstigatorr]

Thank you I just completed my first scrapy project today

[amonged]

I think we need a video where you talk about all the challenges that will face us when scraping like blocking ip or problems caused by sending too many requests.

[stevefox]

Man!, I'm having so much fun learning from watching your videos.

[TheJdB]

When building my scraper, I love to do it on a jupyter-notebook first so that I could separate the request and parse part of the program.

[RonWaller]

Thanks John, I have 2 questions...First, how do you download the HTLM with requests? I tried looking it up and didn't find the solution. Second, looking at the source, what are we suppose to be seeing? I have dont that but not sure what I am looking for.Thanks

[bn_ln]

Thanks for the great content, your channel is an excellent learning resource. May I ask for a starting suggestion for a project that involves authentication and downloading CSV and Excel files.

[Lahmeinthehouse]

Nice video! What do you use for screen recording ?

[daniel]

parsing locally...men....that was it!!!

[alikorloo]

it helped mate. what lib do you recommend for parsing lxml/html? and ofcourse for async request.get (only) and request.post(rarely). minimal libs just to get the work done. in one of your vids u talked about selectolax, and request-html in this one. I only need those two functionalities I mentioned above(parsing, requests). much appreciate it.🙏🏼

[drac.]

How would you recommend dealing with IFrames? Any tips to extract data from those easily?

[BeSharpInCSharp]

Wonderful video. Do you have any on decision tree ?

[chiamaka]

Wonderful videos you have. How can I select the columns I want to scrape. Maybe the the information I need is in column 1, 2 and 4. How do I don that? Thank you

[ahmedgamalelkattan]

We urgently need video about scraping from TripAdvisor using Selenium please 😀

[eziola]

I'm starting to see #shadow-root elements that I don't know how to get into. Thoughts on these?