Python Scrapy Tutorial - Cats & Spiders? Web Scraping Reddit with Scrapy [2020]

Today we’re going to use Scrapy to scrape all the top-voted images from the cats subreddit. Because why not? Cats are great! Web scraping is basically pulling data from websites for use in an outside application. And so, we’re going to use this when a website lacks a dedicated API for pulling data. We’ll write a program to extract the information we’d normally see when visiting a web page.

JOIN our online community of people who want to level up their developer skills ►

***************************************
Timestamps:

0:10 - Intro
0:39 - Kite Promo
1:03 - Core Concepts (Spiders, Items & Selectors)
2:18 - Installing Scrapy
3:38 - Creating Spider
5:20 - Inspecting HTML & DOM
8:41 - Running Code
9:00 - Cat Pics!

#scrapy #cats #pythontutorials #python

***************************************
Additional Python Resources

Read and Write to CSV with Python ►

Web Scraping Baby Yoda ►

Best IDEs for Python ►

***************************************
Be sure to subscribe for tutorials, project breakdowns and more!

STAY TUNED:

[quaidcarlobulloch]

Hahaha so many things were repeated, thought I was having an aneurysm

[heiko]

The most important thing, the xpath command, wasn't explained at all.

[callmezingo]

8:25 You don't have to close the file inside the with context manager, it will do it for you automatically

[fpruijt]

It feels like you are reading from a script and do some QA and remove duplicate audio

[ZacharyGuerrero]

2:10 you had me thinkin there was a glitch in the matrix.

[yajan]

Those HTML commands went over my head.

[nghiepcrypto]

I definately watch every single video of your

[lbbrfarangkiinok]

Why are you calling close() on the file? I thought the the purpose of using with was to avoid having to close your files as they close automatically at the end of the with statement?

[brimmed]

lol this dude sounds like a robot. just reading the script. didn't even realize he had "make sure python3 and pip are installed" twice

[anjaliunnikrishnan]

I followed the entire code. But I got a blank html page. Please explain

[alphamangocat]

Does anyone know why the spider can't be found? i'm in the directory and am calling the right name

[jiwan]

damn, you should use virtual environment for the sake of python, installing directly on the system can cause system conflict.

[Saywhatohno]

great video!!! do we always have to pass 'self' in the function? what does it mean to pass 'self'?

[mattmovesmountains]

I really want this kite add-on but my computer's old and shitty - kite actually said it doesn't work on my old shitty computer; not necessarily those words, but that's how I felt. Is there any other version of kite I can run until I get my new machine? I don't actually say "machine", I say computer but I didn't want the pros to judge me.

[SoulfrikRule]

Which scrapy tutorial do you recommend? I need to scrap many sites getting href and then access each one

[tao]

mine says syntax error in line 3 for "Class RedditSpider"

[dewaful]

i followed all the steps just fine but then all i get is broken images when opening the created html file

[arthurmiller]

LOL Scrapy requires Twisted which seems to require MS Visual C++ 14.0 and still won't install.

[markistheone]

Dude, you need a better mic setup, content is great.

[spyg]

How would I approach the 100 upvotes problem? The data is inside many div tags.