Scrapy From one Script: ProcessCrawler

In this video I'll show you how to use the Scraper ProcessCrawler to run a scrapy spider without using scrapy crawl command. We can use the export feed by utilizing some basic settings. This is a fast fun project to show you just how quickly and easily we can create a scrapy spider.

Support Me:

-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------

# Timestamps
00:00 Intro
00:37 Start Code
01:30 Scrapy Spider
03:12 Parse
07:17 Pagination
08:41 Crawler Process
10:37 Run and Fixing errors
11:47 Output CSV
12:02 Conclusion and Outro

[GelsYT]

I'm reporting youtube for why this channel doesn't have a million subs yet. MUST BE RECOMMENDED ACROSS THE WHOLE PROGRAMMING PLATFORM!

[nadyamoscow]

Thanks a lot! It required some adjusting because there was an error saying the 'FEED" structure has been changed. So I had to make

process = CrawlerProcess(settings={
"FEEDS": {
"my_whisky.csv": {"format": "csv"},
},
})

process.crawl(WhiskySpider)
process.start()


I also had to put initial url to

start_urls = [ < initial url> ]

instead of placing it to start_requests(self) - for some reasons it didn`t want to work. But in the end the adjusted version did work fine. So thank you very much for another great lesson!

[apchihba]

Thank you very much, your videos help me a lot with my graduate qualification work at the university.

[Crwbsko]

Thanks a lot for your videos! They are informative, short and very useful.

[tubelessHuma]

It is very easy and effective way for normal projects. Thanks Dear John🌹

[rastaricky]

Great video! Short, to the point and very informative. I'm guessing I'm a little late to the party, because when I try the program, I get a 403 error. :( Wondering if a proxy is required?

[davidw]

Hey John,
Firstly thank you for all your videos they're really informative.
Just wanted to say on this one you could have set the psize query parameter to be the total number of products and you would have one page with every product on it, if that would have been preferable.

[dileepamadusanka]

Thank you for the giving great video lessons...

[karryliu]

I really enjoy your class. Thank you so much!

[josiahoborekanhwo]

Thank you so much!!!...Your videos has been so helpful...you don't know how much you have made life easier for me😫

[CarlosWong]

Great work John thanks again for your videos!

[dimapugach]

Is it possible to export data right in code to dictionary and use it below?

[sandrodevdariani]

I created this py file inside spider. is this reason of scrapy import problem? there is ModuleNotFoundError: No module named 'scrapy'....

[sivaranjjan]

debug :Crawled (403). The page mentioned in the video is access denied.

[dontdoit]

Can you do a video on how to use visual code studio? Talking about all the extensions you use and why them, debugging aso.? would be great

[TypicallyThomas]

Was kinda hoping to learn how to use the scraped data instead of storing it. Getting some kind of return value like a list of dicts with the data so I could have another function do stuff to that data without saving it

[IanWatts]

Hi John - great series of videos - I'm learning a lot but I have one question. When parse yields, where exactly does it yield to - it seems to yield to the console, or an external file. But what if I want to put it into a dataframe, or some other structure, and pass it into another part of the application for processing. I cannot seem to work out how to create a data structure from the parsed data and use it within my application for further processing, as scraping is just a part of a wider application

[yusufbuyukbirer]

Hello John thank you so much for this video it was very informative. But I have one question. After I run the program there is no data in csv file. Can you help me please?

[jayanthl]

Reactor not restartable!
Any solution ?

[ignacioespinolamajo]

Great video. How can I change the path where the csv file is stored?