Web Scraping using lxml and Python 2018: Extracting data from Steam

Hi everyone! In this screencast I am going to teach you how to scrape data from Steam using lxml and Python. We will be using Python 3.5 but it should work with Python 2.7 as well. It is going to be a fairly short screencast and it is expected that you know basic Python and have lxml and requests installed.

If you don't have the "requests" library installed please install it using pip: pip install requests

After watching this screencast please share you feedback in the comments below so that I can improve future screencasts :)

Have a great day!

[Rey_eo]

Very helpful! Straight to the point and easy to understand! Thank you.

[sayamrahmanrobin]

i am from Bangladesh, very good tutorial. Excited to learn from you again

[kodaio]

Best guide I've watched so far. Thanks for the informative content.

[briansanders]

If you wanted the the script to iterate through all of the pages of games how would you go about doing that?

[Pad]

what about for sites that we are signed in to?

[koops-]

Great Video! Thanks for taking the time to make it. You make everything look so simple!

[bilalshakir]

how to extract number of comments from youtube video as this information not showed in soup

[marykilewe]

First comment. Excited to learn from you

[ramuannamalai]

is there anyway to perform "CLICK OPERATION" by using web scrapping without using selenium @ Yasoob Khalid

[mohamed.montaser]

what is name of this app that you write your code in?

[cKyKTK]

waiting for a video about uploading it and using it as an api

[poojakhatri]

Hi. Thank you for this video.. I have a question.. I am working on scrapping a very large text to csv. While doing this, I get the text in new line after 32755 characters. I checked online, the cell limit of csv is 32k..Can you please give me an idea what can I do in this case?

[виталийсоловьёв-фп]

Can lxml scrap java-script generated part of the page?

[Aasupcollegedays]

how to extract youtube video from web page ?i need help

[rishabhshah]

request,
1 video for django bugging to advance

[tuobraun]

Hi, thank you for the wonderful article.
I played a little bit with the code as per your guide and tried to implement something a little bit by myself:

The code can be run saving the pages but I got this message and can't figure out what I've done wrong as I'm a beginner. Why is it "out of range"?

Traceback (most recent call last):
File "h:\python_scraping.py", line 19, in <module>
if contain_prod_data(data):
File "h:\python_scraping.py", line 14, in contain_prod_data
IndexError: list index out of range

I also noticed that when the cycle breaks this line is not executed:
26. print("Finished saving pages: " +str(page_len))

Thank you for any comments in advance.

[hamzahelmy]

very nice .. just drop the music.. it is distracting :)