How to Login with Python Scrapy (2022)

In this video, we look at how you can log in to a website using Python Scrapy. We look at how to log into more complex websites that use server-side & client-side values in their login forms.

The article that goes along with this video:

Basic Scrapy project used in this tutorial:

The Scrapy Splash tutorial video:

00:00 - Intro
00:37 - Basics of logging in
02:32 - Setting up the Python Scrapy project
03:35 - Creating the basic login spider
06:48 - Running the basic login spider
07:41 - Explaining the Amazon login process & why we need client-side rendering
08:59 - Adding the settings for Scrapy Splash
10:02 - Creating the Amazon login spider
16:41 - Running the Amazon login spider
17:49 - Outro

[scrapeops]

Hey guys! If you have any idea's about websites that you would like us to show you how to scrape, please let us know! Oh and what programing language/framework too - we will be branching out into videos for scraping with node.js and other languages too :)

[huwjarse]

Thanks - well explained, appreciate you sharing.

[BrokenWorlds]

How can I modify it after login in that it will find all possible url and follow them until all found or depth limit and will return all extracted url in an array?
And how can i start your code like c =

[nodariasatiani]

how can we store and pass cookies to the next_function to parse product details page ?

tried to include cookies_dict = {cookie['name']: cookie['value'] for cookie in response.data['cookies']}
and cookies=cookies_dict in scrapy.Request but received following error:

cookies_dict = {cookie['name']: cookie['value'] for cookie in response.data['cookies']}
AttributeError: 'HtmlResponse' object has no attribute 'data'

[awaisahmad]

change password you have displayed in video 😁

[abhishekk]

bro copying a file and deleting entire content from the file.... instead could just create a new file na