Python Web Scraper Tutorial: Sessions, Requests, Cookies & JSON!

In this Python web scraper tutorial I use Python 3 and the Python requests package, to scrape a list of online users from a chat application. In my next upcoming video I will teach you how to scrape Amazon so make sure you hit that subscribe button to stay updated.

To follow along with this tutorial you must have the following installed:

I demonstrate the use of sessions, and cookies, using the python requests package and show you how easy it is to make requests to applications that require login. I also breifly demonstrate how to parse the data returned using JSON.

If you like this video and would like to see more videos and tutorials on web scraping please show some support by giving me a thumbs up and subscribing. Leave a comment with any questions, or requests for future videos.

DISCLAIMER:

In this video I demonstrated a concept using a public website. Please be cautious about making requests to the same endpoints as it may be against the terms of use for that particular website. Use your skills to help the world, not destroy it :)

I write slimmed down code in my video tutorials to make it easy for users of all levels to understand. You don't need to comment on my code. I advise a pythonic code style, and always check for errors! Happy python coding.

Python, Python 3, web scraper, authentication, cookies, requests, tutorial, pip, http, web bot, chat app, python web scraping example, web scraping for beginners, python requests get, python requests post, python http request, python requests library, screen scraping, website scraper, web scraping tools, python web scraping tutorial, screen scraper tutorial

[phibroptics]

A basic lesson, but that is the intended audience. I like the pace, most basic tutorials seem to assume that this is the viewer's first time coding, and not just new to the topic. Thank you for this!

[arete_]

This is interesting. It would probably be more useful to get the cookie with Python programatically instead of copy-pasing it.

[cours]

nice but how do you programatically login and update your session with the login cookie?

[jasongg]

Ive been trying to dive into more python lately, I know OOP and other languages well but never gotten into Python so this was right up my alley.

[EnglishRain]

Amazing! Thank you! :D Please make more such Python videos!

[nancyduhal]

you made it so simple, thank you, i've been searching for a way to do this for days

[tusharmehan]

This is exactly what i needed, Thankyou tons :)

[Quantum_Nebula]

Thank you god. 🙏 I love this, this is exactly what I was looking for. The other top video that came up after searching for this was so confusing. Not to be the hyperventilating nerd... But this might actually be the way I'll be automating my ticket queue at my job ( IT Linux Administrator ). I'll reply to this message with an update on whether I can pull ticket information out of ServiceNow.

[earthwalkerliu]

This tutorial video is really helpful to me, thank you so much. Looking forward to more videos from you.

[edgarramos]

Me funciono, tenga su like buen hombre!

[muntazirabbas]

please make a detailed video on cron jobs (scheduling the scrapper for automatic scraping)
Thanks

[ozkankurt]

Jay Jay Çok Hoş Duruyor, Hemen Kendim Denemeliyim...

[AntonSk]

Thanks for the video. Btw, you don't actually need the json lib here, you can get the json by calling "r.json()" (which using json lib internally)

[metalmasterlp]

Thanks for this, cookie jar was helpful. But what happens when the cookie expires? you need to change the value everytime?

[WeeHoward]

hey Jay, do you know how to post a json with request to login and scrape after login?

[jackroark]

Could you have gotten that cookie programmatically?...say, from the previous page, if it's set there?

[vineetpatel]

i want a to get cookie from login response and use the cookie to next request. How can i do that

[mikkun_]

What if it needs 2 cookies to set?
Cookie: name1=value1; name2=value2;

[Wabuh-Wabuh]

can you make a video on how to auto scrape cookies?

[Daxo_Dzg]

Can u show as how to do multiple requests http at the same time 😢