Python Scrapy Tutorial - 13 - Storing in JSON, XML and CSV

Now that we have successfully scraped data from quote website and stored them in these temporary containers we can finally go to the next step and learn how to store the scraped data in some kind of a database or a file system.

So in this video we are going to be learning how to store this extracted data in a JSON, XML and CSV file.

Next video - Pipelines in Scrapy

#python

[codeoncoffee]

I don't usually leave comments but you are great!!! Love from Iran

[abrahandorado]

first scrapy tutorial i watch that is not a copy-paste of scrapy docs examples and has actual content / explanation..
good job!!
regards from Argentina!

[mutalemulenga]

I am enjoying learning Scrapy from you. You are making the hard stuff simple and you have earned a subscriber.

[Chillos]

God bless you bro! This is truly fantastic!!

[Nirmal_rai]

u r a untold legend buddy. please continue making these amazing videos

[panchomarquez]

You can also put some custom setting in the spider and declare the name of the file and the format. That way you don't have to use the -o flag every time

[mustafapolat]

Finally, ı am able to code and scrape the web page I wanted to use in my PhD studies. Thank you sooo much!

[Coney_island]

bro, you are simply so good! you make the hard stuff easier! thank you very much for the knowledge.

[sowjanyakake]

You are a champ man.. excellent tutorial series

[saurabhhchavan]

Hey man🌟I just say thank you thank you thank you ❤️ my all concept are cleared 😊

[featherx]

Hey sir, I using IDLE as my editor and I am having difficulties getting my files into a json format. I am only able to get one line into the json format but not the rest, I know this has something to do with the yield but I am not sure how to go about it, because I am still new to python.

[mockingspyhd]

BEST PERSON THANKS FOR THIS VERY USEFULL TUTORIALS!

[MinChitXD]

Man, not gonna line, using either requests or selenium will never give me the mood or coolness as using scrapy to scrap the data from webserver, thank you very much

[DanielWeikert]

Explanation is really good. Great work. You really know all the details, respect!
I scraped images from google search. However the image size is very small. Any ideas here? Thanks

[rizaldizakarias]

Hello, does anybody know how to set a delimiter in the cml when storing the scraped data to csv format??

[nasbasarinkm]

in my case the scrapping is working but the data is not writing in to files . why?

[gianfrancodagostino]

Do you have any specialized course? I'd definitely take it!. The simplicity in how you teach is just Awesome.

[arthurh]

But what if I run the script with the little green play button on the top right instead of the terminal, how can I set it up so that scrapy stores it automatically for me in a json file?

[tron_town]

if i don't want to go through the terminal to do the 'scrapy crawl quotes -o items.csv', how would i do that? how can i save to a dataframe?

[cooller]

your course of this helping me a lot, there are even bigger channels out there focusing on scrapy, but the one Indian guy is still the best :D thank you, bro, btw. I'd point out the encoding of the saved result, it's easy to fix the Unicode values, but I'm sure you will take about it later :D