Introduction to Web Scraping with Python and Beautiful Soup

Web scraping is a very powerful tool to learn for any data professional. With web scraping the entire internet becomes your database. In this tutorial, we show you how to parse a web page into a data file (CSV) using a Python package called BeautifulSoup.

Python Code:

Sublime:

Anaconda:

JavaScript beautifier:

If you are not seeing the command line, follow this tutorial:

Table of Contents:
0:00 - Introduction
1:28 - Setting up Anaconda
3:00 - Installing Beautiful Soup
3:43 - Setting up urllib
6:07 - Retrieving the Web Page
10:47 - Evaluating Web Page
11:27 - Converting Listings into Line Items
16:13 - Using jsbeautiful
16:31 - Reading Raw HTML for Items to Scrape
18:34 - Building the Scraper
22:11 - Using the "findAll" Function
27:26 - Testing the Scraper
29:07 - Creating the .csv File
32:18 - End Result

--

--

Unleash your data science potential for FREE! Dive into our tutorials, events & courses today!

--

📱 Social media links

--

Also, join our communities:

_

#webscraping #python #beautifulsoup

[muhammadisrarulhaq]

I was able to make a program for my client i never thought was possible. I got paid real money for this.
Blessings so much learned, this is like magic

[Datasciencedojo]

Table of Contents:
0:00 - Introduction
1:28 - Setting up Anaconda
3:00 - Installing Beautiful Soup
3:43 - Setting up urllib
6:07 - Retrieving the Web Page
10:47 - Evaluating Web Page
11:27 - Converting Listings into Line Items
16:13 - Using jsbeautiful
16:31 - Reading Raw HTML for Items to Scrape
18:34 - Building the Scraper
22:11 - Using the "findAll" Function
27:26 - Testing the Scraper
29:07 - Creating the .csv File
32:18 - End Result

[alternistom]

It's weird to think about it like that, but this video started my whole Python learning back in 2017 and I am SO SO SO much thankful for it.

[adamhemeon]

Two years into a web program and a year working in the field and never bothered to learn how to do this. Great video, I followed along 5 years later in 2022 with Python 3.7.8 and it still works.

[YasarHabib]

This was by far the best introduction to web scraping I've found online. Clear, concise, and easy to digest. Thank YOU!

[IRupalPatel]

I am from commerce background. I have zero knowledge of all the programming language. I found your video and explanation so good that at least now I can start my journey into scrapping and coding. I am so thankful at the moment. Love your channel. Thank you so much.

[evanzhao]

If you had some prior experiences with web crawling, this video can makes your crawling skills into a whole new level. Allows you to crawl website containing complicated info about multiple items into a very organized dataset. The various tools introduced in the video are also fantastically helpful as well. A BIG THANK YOU

[devendravijay]

One of the best teacher I have come across Youtube. Web Scraping explained so well that even a layman can follow and understand the basic concepts. I wish, in life I had a teacher/mentor/friend like the one teaching in this video.

[viveksuman]

I saw this video and then successfully wrote the entire code without looking at the video. Not even once. This is because i understood every line of it. Thank you man. Your explanation is very beginner friendly.

[EustaceKirstein]

32:30, I started cheesing at how awesome the end result of this whole project was. Definitely inspiring - thank you for the excellent guide!

[arjoon]

This was really good content, definitely the best intro to web scraping I've seen. You don't go through it as though you're reading from the documentation, there's more of a flow.

[pdubocho]

The man, the myth, the legend.

You have no idea how much stress and lost time you have prevented. THANK YOU!

[harsh]

MINOR SUGGESTION
As of 10/03/2019, If you are following along this tutorial. "container.div" won't give you the div with the "item-info" class. Instead it will give you the div with the "item-badges" class. This is because the latter occurs before the former. When you access any tag with the dot(.) operator, it will just return the first instance of that tag. I had a problem following this along until i figured this out. To solve this just use the "find()" method to find exactly the div which contains the information that you want. For e.g. divWithInfo = containers[0].find("div", "item-info")

[gavjan]

Very high-quality tutorial.
How to set up everything before running any code is very nice to include, and timestamping it so people who already know it can quickly skip is just much appreciated.
Keeping the tutorial example script and diverse is very welcome.
Writing it from scratch just makes sooo useful for remembering what was where.

I wish other people made tutorials like this... Timestamping is so useful when you just want to look-up that one thing and don't really remember when it appeared.

[Jackjfisodjfjc]

you look like a god when your writing multiple lines at the same time.

[SnehilSinghsl]

I cant believe I actually sat through 33 minutes learning web scrapping, something completely new to me. I was looking for a shortcut but your tutorial was just perfect! :D Thanks for this.!

[frozy]

wow even almost 3 years later this video helped me so much and helped me to make a program that picks a random steam game, this was so hard, but i figured it out, big props to you and this video <3

[sacroultima]

You are comfortable to listen to. Not because you have a perfect pronanciation and a seamless script you are gliding through. You are just talking but not constantly jumping back and forwards. Accurate tempo and personality in your voice.

New subscripion

[saadiyafourie]

Absolute champion, quite possibly the best code tutorial I've ever watched. Oh the possibilities! Thank you :)

[brendanp]

This is the best web scraping tutorial that I’ve found. I’ve been frustrated for hours trying to use other resources. Thank you for making this, your explanations are thorough and great!