Comprehensive Python Beautiful Soup Web Scraping Tutorial! (find/find_all, css select, scrape table)

Practice your Python Pandas data science skills with problems on StrataScratch!

In this video we walk through web scraping in Python using the beautiful soup library. We start with a brief introduction to HTML & CSS and discuss what web scraping is. Next we start getting into the basics of the beautiful soup library. This includes how to load a webpage, the basic commands you need to know such as find & find_all, grabbing strings from an HTML elements, etc. The final section of this tutorial is a series of exercises where you can practice your skills. In this section we scrape a webpage for links, we learn how to scrape a table and load it into a pandas dataframe, and we see how you can scrape & download a web image. Hope you enjoy!

I’m looking into making future videos on more complex things you can do with web scraping as well as other libraries that are helpful such as Selenium & ScraPy. Subscribe to not miss those.

Join the Python Army to get access to perks!

---------------------
Resources used in this video

---------------------
Learn more about HTML/CSS

---------------------
Video timeline!
0:00 - Intro & Video Overview
1:09 - What is web scraping?
3:51 - Introduction to HTML

Using the beautiful soup library (5:29)
6:31 - Loading in a webpage (requests library)
8:21 - Starting to scrape
9:18 - find & find_all methods
16:00 - Finding specific text/strings in our HTML (regex)
18:38 - Select method (CSS path selections)
25:55 - Grabbing the string/text from an HTML element
28:17 - Getting a property of HTML element (href, src, id, class, etc)
29:41 - Code navigation (parents, children, siblings)

Let’s practice our skills! (33:57)
35:53 - Exercise #1: Grab all social links on webpage in 3 different ways
42:09 - Exercise #2: Scrape an HTML table into a Pandas Dataframe
53:09 - Exercise #3: Grab all fun facts that contain the word “is”
57:59 - Exercise #4: Use beautiful soup to help download an image from a webpage
1:04:20 - Exercise #5: Solve the mystery challenge!!!

---------------------
Follow me on social media!

---------------------

Learn data skills with hands-on exercises & tutorials at Datacamp!

*I use affiliate links on the products that I recommend. I may earn a purchase commission or a referral bonus from the usage of these links.

[KeithGalli]

I made a new tutorial building off of the knowledge learned in this video! Check it out!

[BennyHarassi]

Shouts to Keith for giving us all an MIT education without the MIT debt

[apsilal]

I paid a bootcamp for learning. But Keith you are way above all that. I understood the concepts from your video only. I owe you man!! Keep going and please don't stop putting up such videos.

[adrianapetrova]

The last time I tried to understand BeatifulSoup I gave up. You explain it so easy to understand. Thanks for the hard work and the time you spend on teaching us :)

[symnshah]

I have watched a couple of other videos on BeautifulSoup but believe me this one from Keith is the best one. Keith will take you from scratch to a decent level. Thank you so much.

[dhruvrathore]

Please do a Seaborn Tutorial ! like you did with Pandas, Matplotlib etc. I watched all of them, really glad i found your channel. Simple, informative & on point.

[doomimic]

This tutorial was incredibly helpful! Web scraping is something I've always found interesting but just hadn't been bothered to start learning, yet this video made it easy to understand and covered a huge range of ways to deal with potential problems. Seriously can't thank you enough for this video and will certainly be sticking around for any new tutorials you upload.

[ajaykushwaha-jemw]

The Best thing about your tutorial are that you start from scratch and teach basic and explain each fragment of code with concept. Love from India.

[lefu]

Your tutorials are the best, honestly. Thank you so much for doing this.

[ranveersharma]

i am from india . we really dont get this quality stuff here.. so thanks to youtue and you.. for spreading wonderful knowledge.. keep rocking !

[santoshvaidya]

This is one of the finest videos i have ever seen on training. You are an amazing trainer and most importantly you are explaining things in very simple english, also with examples or exercises that would give an hands on experience for

[rahuldavid]

Thank you so much for this wonderful tutorial Keith! Words cannot describe how much I am grateful to you for making this gem of a video that covers everything you need to successfully scrape a webpage! Trust me when I tell you that NOBODY HAS MADE A BETTER VIDEO ON BEAUTIFULSOUP than you!!! If I could have the liberty of suggesting future videos, I would love if you made a video about "Regular Expressions". Keep up the good work and God bless!!!

[ClaireCodesStuff]

This is a fantastic tutorial. When I last tried to learn beautiful soup, we were in the awkward transition phase between python 2 and 3 and every tuturial was in python 2 because they hadn't released code for 3 yet. I learned 3 because it was "the future". Of course, I then wanted to use BS so I had try and figure out what I wanted to do in python 2. I gave up in total frustration. This is a crystal clear guide and now I actually understand how it works and how to use it. Thanks Keith!

[rogueknight]

I wanted to attempt your recent Advanced web scraping tutorial where I then stumbled upon this amazing tutorial and I'm so glad I did! Thank you very much

[kinwong]

CV Update: Web Scraping expert.

Joke aside what an awesome tutorial. Felt so satisfying to get the secret message with what you taught!!
Brilliant work!

[LoganNinefingers]

Keith you'll be the first one I cite when I write my nobel prize winning book or whatever it is nobel prize winners write. Golden content. Gracias!

[TheFearlessGoat]

I love that you have exercises for us to do in the videos! Learned so much from this.

[manuize]

by far the best tutorial on youtube for web scraping. you are very good at dumming it down, even total beginner can even understand.

waiting for NLTK tutorial.
thank you

[benyoyo]

Keith, your videos are excellent. You are totally getting me through grad school just watching your tutorials. Keep it up!

[investandcyclecheap]

This is such a great tutorial ! I loved being able to pause and figure out the problems on my own. I really learned a lot! Thanks Keith, you rock!