Web Scraping : Selenium inside Scrapy | How to login past javascript & send response back to Scrapy

Combining Scrapy with Selenium :

A major disadvantage of Scrapy is that it can not handle dynamic websites (eg. ones that use JavaScript).

If you need to get past a login that is proving impossible to get past, usually if the form data keeps changing, then you can use Selenium to get past the login screen and then pass the response back into Scrapy.

It may sound like a workaround, and it is, but it's a good way to get logged in so you can get the content much quicker than if you try and use Selenium to do it all.

I know Selenium is for testing, but sometimes you can combine Selenium and Scrapy to get the job done!

#scrapy #selenium #webscraping

pip install selenium

[kevinz]

I’ll stop commenting on all these videos but literally everything I have problems with you have a video on. Unreal. Thanks

[MohAmuza]

thanks for that magical line:
res =
we can also do this
from scrapy.selector import Selector
res =

[TheKyaiMusic]

@python360 How do I actively see if the script is working? I'm unable to 'print' the login phase to check if it actually worked succesfully. print doesnt seem to show anything either. Kinda hit a dead end :(

[jamezz]

Hi, im just wondering where do i put that code. is it possible to see an example and learn form it. Thanks

[HafizMuhammadAmirChishti]

Hi @Dr Pi, thanks for the good content. I'm new in web scraping, how to deal with captchas? I'm trying to access a website that is captcha protected, I can't access it without solving it first. Any automatic or manual way of doing it?

[danielyoo]

This is exactly what I was looking for, thank you for the upload.

I'm trying to web scrape instagram to get the dates of each posts (image) that come up for a #hashtag search.

Since you have to log in instagram, I was wondering if you could use Selenium to log in (as explained in your video!)
and then use scrapy to go through each (image). So far I was able to log in...

but, I"m having trouble with extracting the new response variable from Selenium so that I can utilize it in Scrapy.
I've tried using your code at 1:38 of the video, but I seem to get an empty list.

Is there a possible method of doing this?

[learncodeinbangla]

Hello dear,
Your teaching is very good for learnes. I have learn many things from your videos.
But I am unable to scrap a site. That is yellow page au.
Could you please make a video on this site scraping.
I will wait for this video.
Thanks.