Run Scrapy Spiders from Python Script

🔥 codeRECODE.com EXCLUSIVE
⮕ Become a member and get access to all the courses on my site:
⮕ Take the course on Scrapy Basics for $1 or free 😀
(Use coupon code *FREE* on checkout page)

🔓 SOURCE CODE

📠 GEAR I USE and RECOMMEND
⮕ RØDE PodMic
⮕ Audio Interface - Focusrite Scarlett 2i2

📕 CHAPTERS
00:00 The objective
00:25 Path of the script
01:01 Method 1 - CrawlerProcess
02:30 Project Settings and Spider Settings
04:48 Method 2- CrawlerProcess
06:18 Running spiders one by one

Note: These are affiliate links. I get a small commission if you click on these links and buy. This does not cost you anything.
This video covers python scrapy, scrapy shell and web crawler
#python #codeRECODE #webscraping #scrapy

-~-~~-~~~-~~-~-
Please watch: "Making Scrapy Playwright fast and reliable"
-~-~~-~~~-~~-~-

[carloscampos]

Thanks! This video is exactly the answer of the question I had in mind.

[ranu]

Remarkable. exactly what I'm looking for

[RajbirAhmedOfficial]

Let's say I have worked on two templates. With basic spider, I obtained domain names and with crawl spider, I obtained emails. How can I integrate both scripts into one? Then export in one CSV file? Can we do all these things with just one script? Amazing contents by the way 😍

[learndjango]

Hi, thank you .

I just want for you to make a full video to telling me how can i connect between Django with Scrapy

[dariorodriguez]

Hi, thanks for the video. Do you now why some spiders run ok from the spider file, but don't work with the main file. Thanks very much

[raahilbadiani]

thanks for the video. had a a query though. when we run the spiders using scrapy crawl we have an option to pass argument using the -a flag. how can we do that when running from within the script

[francislim]

Hi Sir, Do you have an idea how to sort items before yielding?

[triott]

Hi, great video I managed to run multiple spiders from same script. I'm trying to write a concurrent project with Scrapy and want some advice on if CrawlerProcess is concurrent with Is it better to write CrawlerProcess in separate spiders files (where each will have to initiate 'import CrawlerProcess') and be called from a main.py file with To be more precise will CrawlerProcess overlap each other and not being concurrent if multiple CrawerProcess() calls are being made from the same file (and possibly asynchroneous instead of concurrent) even with Thanks.

[MoVolto]

is it possible to call scrapy and pass a URL so that it does its job and returns the scraped data. Then when you have another URL pass it again to scrapy and get the scraped data back, and so on and so on? I can't make it work

[mvstu]

Sir, are you a professor. The way of explaining is like a professor. If you are, please reply.

[jaeminpark]

Hello, I am running one working Scrapy_spider. Then, when I do, r = requests.request("GET", url), then there is a new message such that "info: recevied sigint after terminate". r = requests.request works fine in a separate python file. It seems to me that scrapy still has an influence on the coming request.. Would it be okay if you could give guidance as to how to run an independent request after scrapy?

[marcon]

Hello! Nice video. I was able to get CrawlerProcess running without issues, but when I try to use CrawlerRunner, I get this error:
Exception: The installed reactor does not match the requested one
2023-12-12 17:06:46 [scrapy.addons] INFO: Enabled addons:
[]
Could you help me please?

[pythonically]

when I create a virtual env in windows I don't see the venv folder in the root directory of my project. but when I do the same in WSL it create a folder (venv) in my project root directory.. why?
I use miniconda to create venv by runnig this command conda create --name venv

I think in windows it get venv get stored somewhere else? am i right ?

[shashwatpandey]

Hi, I love your videos. I learnt a lot from you. Thanks for your videos.
I had some problem while scraping a website ticketmaster, would you please make a video on how to do that because when I run this website using scrapy it can't even connect and throws some forbidden status.

[harikrishnanv]

Sir, can you show us scraping an unstructured website, with try except and conditions. Because most of the sites have different patterns on different pages. Thank you

[hamzaehsankhan]

For some reason, I was getting the error:
Exception: The installed reactor does not match the requested one
I looked into settings.py and the following was uncommented:
TWISTED_REACTOR =
I commented it and using the reactor and defer to run spiders sequentually, worked.

[pythonically]

was looking for this video for a long time I think it was not possible to run multiple spiders from one script and lose my client because of this ignorant. but I am just facing a single error which is that I am not getting any output.csv files in my directory. although both the spiders are throwing perfect results in terminal.. why ?

[kumaranmuraleetharan]

Awesome video, thank u🙏 Could you make a video to convert the same spiders to run as an exe file using pyinstaller or anything similar?