Web Scraping Databases with Mechanical Soup and SQlite

Hi Everyone! In this step by step tutorial, we will extract a huge table of data from the internet and store it inside an SQLite database!
To keep things simple I've chosen a Wikipedia table, but I highly encourage you to apply the same principles on data that updates a bit more frequently (for example weather forecasts) 😃

⭐clone complete tutorial code⭐

⏰ TIMESTAMPS ⏰
***************************************
00:00 - intro
00:34 - imports and installs
01:42 - web scraping with mechanical soup
02:20 - select HTML table elements
03:47 - extract element attributes
06:11 - find the index value of a list item
07:13 - extract multiple columns of table data
09:44 - organize extracted columns
12:44 - enumerate function
14:02 - dictionary to data frame
14:53 - create SQLite database
15:36 - create SQLite table
16:35 - insert Pandas data frame into SQlite table
17:26 - save data permanently inside database file
18:49 - thanks for watching!

💻 CODE AND IMPORTANT LINKS 💻
***************************************
⭐ URL used in the tutorial:

⭐ complete code repository on Github:

⭐install SQLite on Linux:
sudo apt install sqlite3

⭐install SQLite on Windows:
Download the Precompiled Binaries for Windows zip file from SQLite docs:

⭐install SQLite on MAC or Anaconda:
no need to install - you already have it! 😁

⭐ code used in the tutorial:
column_names = ["Founder",
"Maintainer",
"Initial_Release_Year",
"Current_Stable_Version",
"Security_Updates",
"Release_Date",
"System_Distribution_Commitment",
"Forked_From",
"Target_Audience",
"Cost",
"Status"]

📽️ RELATED TUTORIALS📽️
***************************************
🌞 Much Better HTML table Web Scraping with Pandas:

🌞 SQLite Databases for Beginners:

🌞 Web Scraping Images with Mechanical Soup:

🌞 Web Scraping Text with Beautiful Soup:

[inconnumj]

can mechanical Soup scrap dynamic content "javascript pages" ?

[VisuallyExplained]

This is probably one of the best and most thorough channels related to practical applications of Python. You have a very unique style, very well done!

[chaghlarblabla]

As a blind programmer i find it very useful when you tell us what You typed. i subscribe Your channel now. Keep it going.

[TheDroc]

New subscriber!! Im a Data Engineer with 10 years of experience. Can't wait to watch all your videos! 👏

[CrypticPulsar]

You are just too awesome! Simple, powerful, and fluent method to easily remember commands that you wouldn’t otherwise.. bravo, and thank you!

[giorgosiotis]

I wish all female minds worked a little like you. Congratulations from Greece

[shadyabdelhady-rmsz]

thank you so much, that's exactly what I'm looking for

[stay_stoic_be_stoic]

When you said simplied, you are on the money. I needed to get a clear view on OOP, your video literally cleared all my doubts.

[Pradeep-kvhp]

people call Python is easy language but you are makes python so easy and understanding by tutorial videos. Thanks and keep it and make it more and more tutorials videos

[wslater]

wow - i am very impressed - so much easier than going to coding sites where the logic is harder to translate. Thanks for clearly explaining the logic within each line of code

[martinmiguez]

Ohhh magnifique!!! Love love love this tutorial!!
your work always improves. The way you explain and how you use the logic of programming is very clear.. thk you

[urbaneplanner]

Nice walk through - there are many ways to webscrape - I hadn’t come across mechanical soup before (I’ve used beautiful soup, selenium, and scrapy and I would approach this task a bit differently) and I really like this intuitive walk through as the thought process applies to the various programmes

[jefferyandme]

I'm hooked on this channel!! Easy to look at and her style resonates with me. She has a real way of helping me understand. Very refreshing!! Thank You!

[alisheik]

Hi,
You have an unique way to explain the subject. I am really impressed the way you explain. I have seen hundreds of videos about web scraping, but none of them as simplified as this. Thank you so much. I heard about Beautifulsoup, Scrapy, Selenium, Puppeteer. But for the first time MechanicalSoup. And I am expecting some more videos about web scraping with the updated versions. You are techings are as sweet as your voice.
Thanks

[tonym]

You have a great tallent to teach us complex programs in the east way. 👏👏👏🌻☘️🐱.

[kamertonaudiophileplayer]

Now I better understand how Google works. Indeed, it is a simple as a soup. Thank you.

[dravidaravindkumar]

Maam your teachings are awesome 👌👏..can you plss put a video on stadium seat booking system with python as front-end and SQL as backend...

[bc]

This is exactly the topic I was looking for, but it was crazy hard to find, so thank you!

[Jason-siyd]

Great job on presenting this. It was kept basic and very informative.

[zparihar]

Definitely the cutest Python Teacher in existence!
I'm officially giving up Ruby!