How to scrape INFINITE scrolling pages using Python and Selenium (2 Methods)

In this video i show you how to scrape infinite scrolling pages using Python and Selenium with 2 different methods. Knowing how to scrape infinite scrolling pages takes you one step further in your python web scraping journey. Twitter is one example where you need Selenium to scrape infinite scrolling pages. The first method uses the execute_script function in selenium and the second one uses the send_keys function.

If you have any questions about scraping infinite scrolling pages, feel free to leave a comment down below.

[woahrajveer]

I cant thank you enough for this amazing explanation!
someone else had also used a similar method for scrolling till the end but i didnt quite understand what they had done and you explained it brilliantly!

[GabrielCastro-muqr]

sir you really saved me here. I applied this logic to my own problem and worked.

[amritapattnaik]

Hi..Thank you for this amazing solution. I have been searching and searching but wasnt able to find many on the endless loading of pages. Thanks again...

[richardtvtv]

This has saved so many manhours for me. Thanks mate 😀

[krsnamay_nihon]

Loved it brother. Thanks for sharing this.
I would love to get more videos from you on handling lazy loading

[ToucHDowN]

Great tutorial. I was wondering, what fix can you do with the [send.keys] method in order to stop the script, like you did with the previous/new height in the first method? I see that the script keeps running for ever even if I reach the end of the page, thanks!

[advertemp]

So Lovely !!!! <3 And i Love Your intro it's So Honest ! So Creative !

[moudhafferbouallegui]

Can you set a condition so that before it scrolls it tests for the existence of a button, if the button exists it clicks it and continues scrolling otherwise it just keeps scrolling?

[abhishekpandey]

i am trying to scrap the data from zomato and tried both methods but in first one my page is directly scrolling to the page end and in method 2 I am getting this error :-
AttributeError: 'WebDriver' object has no attribute 'find_element_by_tag'

[rishabhjoshi]

now the command is:
driver.execute_script("window.scrollTo(0,

[swathireddy]

Hi how do we scroll inside a container which has virtual scrolling in python selenium

[milagicheva]

How it can be implemented with 'load more' button?

[henishshah]

This is so helpful. Thank you so much!

[alessandr]

Great content !!
One question, how do i parse data from every page while scrolling ?

[landrew]

Fantbulous!!
The code is available on twitter, but not work on youtube page. Do you know why?

[umeshchaudhari]

This is much needed sir and appreciated on top. But I have a question like how can we scroll down in nested scroll. Suppose we have table on a page and we want the data from that table which generates html after scrolling the table scroll not the page scroll.

[ProjectSkillsQMUL]

Thanks for the video. Do you know how to scroll the page partially and then keep going to the end of the page? The website I am trying to scrape doesn’t charge the products if I scroll down too quickly using either methods so I can’t get all the products.

[abdulwali]

hello bro can you please tellme how many requests can i send on a website pages without being blocked? like how many in a minute? a safe way if i dont want to use proxies.

[PEV]

Can anyone tell me what kind, of course, gave you knowledge of this handling of web pages? It always feels like I am lacking something in this, I am looking for a comprehensive course around this topic.

[kritishah]

Sir What’s the difference between the Selenium and SelectorsHub?