Web Scraping with Python - How to handle pagination

This is the second video in the series of scraping data for beginners. We're gonna to really clean up our code by adding functions and adding support for pagination, including how to break out of loops.

This is a series so make sure you subscribe to get the remaining episodes as they are released!

If you are new, welcome! I am John, a self taught Python (and Go, kinda..) developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.

:: Links ::

:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

[oneashen]

Love this series man. I really hope for the advanced series too. Thank you for sharing!!!

[Levy]

videos everyday??

oh man, thank you for your time!!!

[MrBenStringer]

Absolute legend. Amazing content. Learning a tonne, thanks dude 🙏.

[deeperblue]

Really valuable for all. Especially when new to this topic.

[thebuggser]

Great presentation! Neat use of Python’s yield.

[AliceShisori]

thank you for this series, I think you should structure your future videos like this too. so maybe complex ideas/projects will be displayed better.

you got a course or something on udemy? I'd love to buy it both to learn from you and to support you a bit to show my gradtitude. I don't have a visa or credit card so I can't thank you on youtube!

[sifar]

Maybe if you could show how to pull all pages by handling how to bypass rate limit & ip blocking using rotating ip/ user-agent etc, then it becomes interesting! Hope you add such videos to this playlist.

[WhiteFontStudios]

REI Shop: "Why is our conversion rate 100, 000x lower on Camping and Hike Deals??"

[zakariaboulouarde]

Thaaaank you so much, helpful 🙏🏾🙏🏾. You're the best.
Do you have any recommandation where we can host like this script as an api with fastapi framework or flask?

[muhammedjaved]

Your videos are always amazing love your way of teaching can you please make video one google map contects scraping

[danlee]

Great video as usual John.
Per your other videos, would finding out max page count be alternate way for pagination stop condition versus checking for not 200 OK http response code? I like how you showed this option though. Thanks.

[Omarwaqar-ptwf]

If we scrape a website let's say every hour generally speaking is there a chance that we'll get our IP blocked ?

[SivaSakthiRajagopal]

Can you make a video to scrap the data from tripadvisor restuarant ?Like a big website

[itumelengmadumo]

How would ou go about a webscraper that monitors changes to a website and notifies you ?

[Omarwaqar-ptwf]

Would love to see advanced web scraping with puppetier

[umerjavaid]

I am learning alot John But i would recommend to make it more advanced level i had texted u at twitter too.. it would be of a great help if you make a complete series related to scraping explain each n every aspect used in modern day scraping

[Fabricio-mquk]

John, could you tell me why httpx works with some urls and not with others?

[KontrolStyle]

Thanks for lesson. I keep getting "NoneType" error -- "AttributeError: 'NoneType' object has no attribute 'text'" - on 22 in video - but it still runs through with the code. if I just keep hitting continue. 😄

[vinodbabu]

can you make a video on how to use neovim

[Антмара]

Hello John. Thanks for your videos. I’m learning scraping and recently saw one order on freelance, I decided to complete it for myself (to test my knowledge). The problem with this task is that when there is more than one page in a category, the site only returns data from the first page. 72 products are posted on two pages, and when you collect information from two pages, you get 36 products that are duplicated. I think the site has parsing protection. but how to get around it? I use a random proxy and user agent. What do you think about this? Can you give me your hint, what is the matter here and how to solve this problem.