Easy Way to Handle Form POST with Python Scrapy

Handling POST requests can be a little tricky with scrapy. In this video, we will see a specific scenario, where the POST request is a submit form to the same URL. Especially useful when working with the ASPX page.

🔥 codeRECODE.com EXCLUSIVE
⮕ Become a member and get access to all the courses on my site:
⮕ Take the course on Scrapy Basics for $1 or free 😀
(Use coupon code *FREE* on checkout page)

🎥 RELATED VIDEOS

🔓 SOURCE CODE

📠 GEAR I USE and RECOMMEND
Note: These are affiliate links. I get a small commission if you click on these links and buy. This does not cost you anything.
⮕ Røde PodMic
⮕ Audio Interface - Focusrite Scarlett 2i2

📕 CHAPTERS
00:00 Introduction
00:30 Analyzing the Site
02:57 Spider for the First Page
06:12 Working on Form Submit
07:12 Headers
08:02 Form Data

#webscraping #python #codeRECODE #upendra

-~-~~-~~~-~~-~-
Please watch: "Making Scrapy Playwright fast and reliable"
-~-~~-~~~-~~-~-

[stg]

I've spent the last two days trying to figure out why my POST requests are turned down. Countless UDEMY research and re-study later, found your answer. Simple, but extremely important. Thank you.

[tubelessHuma]

Its very important lecture like your previous on on POST request. Thanks a lot. ❤

[teodortodorov]

Thank you for taking the time and providing this information. Much appreciated.

[Seblful]

Thank you very much! It is too helpful for me

[BeBold_]

What if i have to pass specific user agent? Should i have mention it on every steps/tabs in code?

[chaimryd]

Thanks for the video, it helped me a lot.
Could you maybe help with cases of PostBack?

[lremipsum]

Hi, question, any tips/how do you select which fields are important to be added in formdata?

[A__GARV_SAXENA]

sir how to access terminal :_) bcz im usign default idle thats comes with python

[cosmicblack]

Good video and thanks for share
I have a question,
Im scraping a jsf page to extract some notes. The problem i have found is that the page has a link for each page but in his @href atribute just appears "#" and, above this a script element to hide the info
There is no XHR requests jus javascript.
Other words, the info is hidden by js and i cant find how to "decrypt" the info to get the links (in the <script> are sent some info"

May you make a video for scraping jsf pages ?

Anohter questions, about asyncio, i dont understand the documentation of scrapy to install it or activate. I cant use playwright with scrapy because of some problem with asyncio but i dont know hot to insall

thanks :)