Scraping multiples websites with one Python script

Writing a simple web scraping script to do some basic price comparison

Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases

[zhengdiao]

Learned a lot in your video, hope to come out with a neovim editor tutorial, thank you sir!

[silkogelman]

Thank you John! 🙏
Informative and it got me a couple of new ideas I want to try now! 💡😀

[bainsk]

Great video John, thank you. Very informative.

[sheikhawais]

Could you also make one tutorial for your code editor setup and the terminal? It looks really cool.

[shehbanpatel]

Hello, I tried this but keep getting the Attribute error 'NoneType' object has no attribute 'text'. I outputted the text this resp receives and it doesnt have the tag which shows up while inspecting the page

[dennistanui]

Thanks a lot, always informative. How would you then run the two scrapers concurrently? and how would you pattern match when scraping a lot of products (i.e scrape all products on both sites, and then create a product_dataframe for example with price comparison)

[mmemahmoud]

nice work as always, can you please make a video about how to scrape email addresses from a domain ?

[lasangagamers]

i have written the code but it will not print any results

[ericxls]

Very good video as usual! Thank you! When is chatgpt video coming 🤔?

[yawarvoice]

Hi,

I"ve asked this question in other video of yours as well, but asking here again, in-case you have missed the other one:
@John I've been following you for a long time and watching all your scraping videos with Python. I have started to create scraper but the website is not allowing me to access as it is considering my script as a bot, though I have changed the user-agent to latest chrome but still, that website is recognizing me as a bot. My question is that which combo I should use for scraping little complex JS/AJAX/bot-aware websites? People say that selenium is good for that purpose, but you say that selenium is not a good option now a days as it is slow, then what do you suggest, which combo should I use, that can fit in many scenarios, if not all.

Looking forward!
Thanks.

[pypypy]

Great video as always! But how do you happen not to be banned by Amazon? I tried scraping a couple of years ago - it always detected my script as robot and didn't give data.

[samoylov]

Thank you for this video! Works wonderful with a particular item. But what if I want to get multiple items. Say, news stories from a website. - css_first gets only latest one. css_all - doesn't work, and just html.css(selector) won't work either. Please help.

[Wassilvideos]

Hi John I have a question, can you guide me for how to scroll down a scrollable ul list in a section of the html with playwright

[herrpez]

Oops. Misspelled Thomann; better remake the video! 😉

[gh-sbdy]

vids they are great
When getting info from a site using python is the ip same or when using python? or do they have their own different ip address? and also same with scrapy; if i use scrapy does that ip address is same as this computers?
because some sites have blocks set up to prevent types of things like this and i dont want to get banned forever by my ip
any way to bypass this so you dont get banned?

[ChristopherBrown-bjzl]

7:05 Yeah but, show me the ugly as sin CSS selectors/HTML. Those are the ones that give me the hardest time. Great vids! Thanks!

[void-qyov]

Hey man, 10x for your tuts.
I'm doing a lot of scrapping. Lately I need to get logos of 20k e-commerce stores.
Imho, it was an interesting task. Unfortunately only about 1/3 could be automated - I went with finding divs, classnames, and image sources having a 'logo' in it.
May be you did something like that before and have interesting strategy ?

[gh-sbdy]

Can you please post your codes in your videos to a link below or in github or etc. it would be so helpful

[c__ne]

Nice! Is this neovim? Can you write to me how to get this editor with syntax highlighting tabs etc? Thank you!