Working with COOKIES and HEADERS in Python SCRAPY framework or REQUESTS package

Hey what's up guys, in this video we gonna learn how to use cookies and headers along with scrapy spider's requests or the bare python requests library and what's the difference between the two approaches.

Source code (scrapy framework)

Source code (requests library)

How to pass cookies within headers in scrapy

[azertyu]

thank you man you saved me and made my day, i never got in mind that scrapy dont parse cookies they mentioned this on the documentation but they dont explained it well

[aarons]

Good tutorial on cookies! Haven't seen a lot about that side of scrapy before.

[python]

Nice comparison man.
Keep 'em coming.
More comparisons if you get the chance!

[tubelessHuma]

Dear Maksim very informative regarding cookies. How we come to know that for which sites we should use cookies in our request?

[frankgunseor]

Hi Maksim, Great job on Cookies, especially the cookies within headers in scrapy. Is this technique applicable to passing Payloads and other 'required' headers such as app-id and say X-XSRF-TOKEN for example? Thank you for a great job by the way! Frank

[davyroger]

Thanks nice tutorial. Also, why not just use an IDE?

[samirbhattarai]

Great tutorial. Can you also make a tutorial about saving and loading headers and cookies to and from file or database using CookiesMiddleware class that scrapy provides? That would be even robust. Thanks in advance.

[mr.himalayan]

I'm from a non-tech background and I'm trying to make some special things in excel.

I learnt from the Internet how to import live data from the site to excel which is update every 2 mins daily for 6 hours of related to stock market, but cookies expire every one hour of that site and I need it to automatically updated on my specific excel cell with anything or using Python.

Is there any way to do this, how can I import cookies from the site and update them automatically in the excel cell.

It will be a project for you also.

[Sergio-tdmn]

Privet Maksim, thx for the great tutorial! The only thing, I am not really getting is how to start using requestbin. Do you have a video on that or something?

greetings from Germany

P.S.: some context. I tried to scrape another site with scrapy and for the first time I got 403 error which I couldn't solve with spoofig headers or setting robots.txt so my guess was that this is a cookies issue.

[arielnovelli]

Hey! Thanks for the video.
I guess in cookies_scrapy.py line 91 should say: url_rightmove instead of url_request, Am i right?

[Deepakkumar-lszm]

Hey bro. Can you make tutorial on scraping data from trip.com website