Scraping Professors Contact Information Using Python

In this video, I’ll walk you through the process of scraping faculty data from Stanford University's Computer Science department website using Python. 🧑‍💻 We’ll leverage the power of httpx for sending HTTP requests and BeautifulSoup for parsing HTML content. The goal is to extract valuable information like faculty names, titles, and profile links and neatly save it in a JSON file. 📂

The tutorial covers the entire coding process: 1️⃣ Sending a GET request to the website using httpx. 2️⃣ Parsing the HTML response with BeautifulSoup. 3️⃣ Iterating through HTML elements to collect data. 4️⃣ Handling cases with missing information gracefully. 5️⃣ Storing the collected data in a structured JSON file.

If you want to follow along with the video, you can check out the detailed article on my website. 💡

If you find this tutorial helpful, don’t forget to like 👍, share 🔗, and subscribe 🔔 for more awesome content!

00:00 - Intro and What we are going to do?
01:16 - Checking website and importing required libraries. How to use HTTPX python
02:30 - How to find data in network tab? check HTML elements using inspect button
04:45 - What is timeout in sending requests? How to set timeout in HTTPX?
05:55 - How to extract website source code using python?
06:10 - Initializing bs4 for parsing data and converting data to html format
07:30 - How to parse data using BeautifulSoup and make it clean
15:15 - How to combine urls in parsing?. adding domain to the beginning of string
25:29 - How to save scraped data in json format? save extracted data into json file

#webscraping #beautifulsoup #automation #httpx #dataextraction #leadgeneration

[SharpDream_]

Can you please make a video on scraping Google Maps with Scrapy?

BTW you are the best practical web scraper YouTuber I have ever seen
and don't think you are bad at explaining you are so good at it
keep on!

love from Bangladesh❤❤

[abdulrafay]

keep making these tutorials :D, always love it

[selmanavaz]

I wish to become an expert python developer like you 😅