Automate the Boring Stuff with Python - Part 13: Web Scraping

Learn Python programming to automate your daily work, e.g. web scraping, working with Excel documents and Google Drive.
We cover chapter 12 of the book Automate the boring stuff with Python by Al Sweigart - covering web scraping in Python.
Follow along to learn Python programming and make your life easier with automation.

Song: The Uneven Path - Trevor Kowalski

Timestamps:
0:00 - Introduction
0:32 - Project: Easily show addresses in Google Maps with the Webbrowser module
04:12 - Downloading Files from the Web with the Requests Module
08:19 - Parsing HTML with the Beautiful Soup Module
12:16 - Project: Downloading all XKCD Comics
14:46 - Controlling the Browser with the Selenium Module
18:35 - Practice Questions

[brandonparker]

Loving this series! When it comes to Python and web scraping, how often do you incorporate tools like HasData into your workflow?

[BillyLongshot]

the Google search project, not presented here, does not work. This I think is because Google is a partially dynamic page. The tree you get in the browser is not the tree that is retrieved by bSoup4. To overcome this, I searched for a pattern of elements like 'a h3 div', I think, which returned my main searches. I then walked the tree with .parent and extracted the 'href'.

[tomkmb]

Working through this book has thrown up more errors than any other beginner resource I've been using. It's beyond frustrating. Some are to do with running Chrome instead of Firefox, some are trying to run scripts through Pycharm and the path situation not being found (even though the directory is in path). Then when using selenium the is getting a traceback saying that webdriver object has no attribute 'find_element_by_link_text'.

Honestly this book gets recommended so much as a beginner resource but I would say that PCC is by far and away the more superior introdution. Working through this as my second book has still be frustrating as hell and nowhere near as rewarding as PCC.

[hazeltucker]

I wish you would pause some between talking so fast.