Hacking With Python #12 - Image Scrapy Bot

This is tutorial on how to use scrapy to write an image scraping bot to scrap images off an image hosting website. All Links and Slides will be in the description. Subscribe for more cool stuff!

If you like what you see be sure to subscribe and thumbs up!

[OMullanRyan]

I stumbled upon this video looking for tutorials on Scrapy.
Really nice video. Keep up the good work!

[RayByrd-ig]

Great walkthrough! How do you think HasData fits in with scrapy for image scraping projects?

[kjlw]

Great Video!
Thank-you!!
Video-idea...downloading images from multiple different pages on the same domain...just a passing thought, from a project I've been working on.

[almjhoolGOLD]

hey Draps :)

Maybe I have some question ;)

what are yield do?, are there another func which do the same purpose?

And just I want to ask another question..

what do you feel about the GUI python programs? Tkinter and another things...


and Thanks for your tut :)

[ciaranmcconaghy]

Great tutorial. Thanks. I can run the standard piplelines version ok. The images get downloaded. However when I try to implement the custom pipeline to get the titles of the images (copied and pasted your code) I get an error at the return line - saying "list index is out of range". The "full" images folder is not created when I run the custom pipeline. Any thoughts why this might be? Thanks!

[alv]

did the title identifier on the site change? Inspect element shows different than in video.

[astronomyy]

Hi, thanks for you guidance, I ran this spider on Windows and it was just crawling pages without scraping anything.What's the problem?

[balaveeraraghavareddy]

Your tutorials are very interesting please do more...

[utkarshnn]

Sir if you could help me out. I am trying to scrape images from a site called zomato But i am failing to do. I have used your code. I have even tried all possible xmls. Please help me out if you could guide me

[nonewsman]

Very clear and informative. I wish you had taken some time explaining the customization of the images pipeline

[amirul]

It was strange for me when I can't scrape the image title (even after getting the xpath using firebug myself). Turns out the javascript running on that page somehow messed up the xpath. So just toggle javascript off temporarily before you get the xpath. I use an add-on to toggle javascript.

[halcyonramirez]

Is there a way to use selenium to make the pages load and extract those pages with scrapy?

because even if you load thee pages with selenium only the initial response will be seen by scrapy. Is there a way to somehow update that response to make it include the new loaded pages?

[fabiowol]

Awasome video !!!
Thank you so much!!
from where can i learn scarpy like that??
Thanks!!

[ShadowHenry]

Hi, I always get 'IndexError: list index out of range' and the images cannot be stored in the local file.


How can I solve this problem? Thx!

[moviesdaily]

There's an error is happening that "no module named imgur". how can i fixed it?

[tiannnanzhang]

I use windows and I type the same code as yours.
Why I cannot download the photo?

[sparksolutions]

Hello...thank you for this tutorial..just wanted to know that when I am running scrapy crawl imgur command, its displaying mkdir Permission denied...please help

[jesa]

hello sir i found this error,
File "/home/jebastin/imgur/imgur/pipelines.py", line 12, in set_filename
return'full/{0}.jpg'.
IndexError: list index out of range

[akashnegi]

first of all great tutorial man but i my spider is giving me error "exception.IndexError: list index out of range"
This is happening here when " item['image_urls'] = ['http:'+link] "
Please help me man.. thanks in advance

[brothamo]

I'm using scrapy version 1.0.3 and when i type "scrapy crawl imgur" it says command not found. Is there any solution?