Python Programming Tutorial - 25 - How to Build a Web Crawler (1/3)

[krishanbhadana]

youtube needs people like you, seriously there are very few people who are sharing their knowledge with the world in such a beautiful manner

[thenewboston]

Just uploaded a crap load of new tutorials. Didn't mean to blow up all your sub boxes. Don't worry, I'm done now. 

[melvinvijay]

what is the difference between

2)request.urlopen()
3)requests.get()

Because in three tutorials you have used three different ways to access the url or webpage.Why can't we use the same request?
Thank you.

[greatsea]

9:08
"I don't know. I don't know if you guys even know what headers are but, there's extra stuff that like your'n...umm...well, its like extra crap that...iii...the user doesn't need to know about truhm..." -- classic Bucky :)

[lich]

install BeautifulSoup and Requests at windows command:

>python -m pip install BeautifulSoup4
>python -m pip install Requests

[channelnamepending]

Holy shit.. i just watched this video, read the Channel name and immediately recognized you as the channel teaching me HTML a few years ago.. thanks!

[ewliang]

This tutorial gave me a good idea of what my next personal programming project should be. Thanks for the tutorial!

[HYPED]

Hey Bucky, I'm able to find this website and it keeps taking me to other pages. Can you just please give the link for that web page that you are working on in the video. Thanks!

[goadsaid]

1. Does anyone know of an acceptable website to crawl? The classes on say, ebay, are not as self-evident as "class = item" and/or they are blocking me from crawling them

2. Is say "max_pages" a built in parameter? Like does python know what you mean without defining it further? Like does it think page 20 is "max_pages? or 50? I had this same question a few tutorials back with "csv_url" when writing the reader because - how does the program know which csv-file containing url you want to open? we passed "csv_url" into the function without ever saying okay csv_url = goog.fgf.csv etc. Did it just automatically assume you wanted to open the link above the user-created function because it was there?

[elderroot]

Haven't finished watching your series yet, but you are using pycharm. So I like you already.

[cangri]

Hey Bucky! What do I do when at the moment of selecting another page from the website, the url is not altered? Do you understand the question?

[Phoebusjosh]

Which sites can I send web crawlers to now that the link posted isnt working

[bjoernjuergensen]

Hi there, using your wonderful spider I have created nices results. How can I export them to csv now? i have spent the whole day to find a tutorials at YT that cover this issue. Can you help, please? Thankx! Björn

[xingyubian]

Does pycharm community works for the web crawler?

[morgohill]

Wesbsites wit rules about what they want crawled and what they don't list this in a file called robots.txt that can be found at the route of the site. There is a syntax to these files so you can make you bot read it.

[preppy]

I dont know why but I cant import requests
it says no module exists

[edwinmurimi]

6 years later and i am still looking for Bucky Roberts....meehn i liked that dude...

[noeramirez]

there are no items showing on your website

[iankavanagh]

Having issues with this error - any idea ?bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?

[leolrg]

what if you want to crawl a website that url doesn't change when you change a page