Read CSV/EXCEL into Python Scrapy - Loading Dynamic Data

In this video, we will learn how to *READ* CSV and excel files in Scrapy. This is useful in scenarios when you want to create URLs dynamically, or maybe you want to read data to be searched from CSV file.
If you like this video, subscribe and take the free Scrapy Crash Course:

-~-~~-~~~-~~-~-
Please watch: "Making Scrapy Playwright fast and reliable"
-~-~~-~~~-~~-~-

[black_metal_luvr]

You brought me to a solution for the issue I was having with the .csv header - Scrapy wouldn't fire from .csv because of the header, and rather than me trying to remove it, this allowed me to utilize it in a better fashion than I was originally going for. Thank you!

[legallyinsane]

very useful easy way of explaining important stuff ... cheers :)

[zeesh]

hi, thats an awesome tutorial. my use case is a little different. i want to scrap through an excel file instead of a webpage. how can i tweak this code to search for some keywords in an excel/csv file?

[python]

Nice tutorial. You might be interested in making your xpath selectors shorter, for instance at 6:36 you could have done something along the lines of :


The default xpath suggestions from developer tools can break / and are harder to find when troubleshooting.

Dr P.

[surajsinghrajput]

What editor are you using sir, it looks very user-friendly.

[somitragupta]

Content is very resourceful.
I want to ask a question on how to apply error handling using callback so that while retrieving multiple files from site using scrapy it wont shut down process if some timeout from server occur.
May be as an idea on future content :-)

[KhalilYasser]

I noticed that the output is different from the tags order in the excel file. I think this is because the first response that is ready parsed first. Am I right? If yes, how can I put delay till the response completed then the code moves to the next request???

[ajayyadav-ushd]

How to read data from database in scrapy ?
where does i mention my database credentials