Scrape ANY Data from ANY website in Home Assistant - TUTORIAL

Sometimes there's no integration for the specific data that you want to show in Home Assistant. Next to the default scrape integration in Home Assistant, there is a HACS integration that allows you to scrape websites and load data from those websites into custom sensors. In this Home Assistant tutorial, I show you as an example how to scrape the current energy prices from your energy provider's website and the top three box-office movies from the IMDB site and how to show them on a dashboard in Home Assistant. Let's do this!

Links referred to in this video:

⭐⭐⭐
⭐⭐⭐

Music by:

00:00:00 Introduction
00:00:51 Example 1: Energy Prices
00:00:59 Example 2: Top Box Office Movies
00:02:12 Install Multiscrape
00:03:05 Install a text editor
00:03:41 Prepare the files to create our scrapers
00:04:26 Explain how Scrapers work
00:06:09 Create the scrapers for energy prices
00:10:06 Test the Energy price scrapers
00:11:31 Scrape the Top 3 Box Office Movies list.
00:12:50 Scrape the title of the movie
00:13:04 Scrape the other values of the Movie
00:14:32 Test the Top Box Office Sensors

#homeassistant #scrape #scraping #scrapewebsites #scrapeenergyprices #smarthome #homeautomation

[SmartHomeJunkie]

What data do you (want to) scrape from what websites? Let me know in the comments!

[frason]

Great video! Made a scraper for swedish namedays. A tip for getting the correct selector in the inspection view (which I had some trouble with): right click the selector you want to scrape, pick Copy and then Copy selector. Then you get the selector correctly. Like so: body > div.header > div.text-vertical-center > h1

[LiamCook-bb]

Wow, this tutorial is so handy! Does anyone know if HasData can be used to scrape energy prices too? Curious about mixing it with Home Assistant.

[value]

Hi Ed
This is a wonderful tutorial! Very clear and step-by-step with good examples. Thank you so much!

[DennisSchmitz]

That worked better than expected. Added local sea temperature as a sensor, perfect for swimming season!

[RobinDeBecker]

Enorm bedankt voor je video. Het heeft me geholpen de scrape sensor eindelijk werkende te krijgen en heeft me meer inzicht in deze sensor gegeven. Een dikke pluim en duim omhoog voor je goede werk :)

[bat]

Please do bring more multi dimension of HA with different use cases like these

[huebui]

Thank you. You're an excellent teacher. I've learned a lot from your videos. 👍

[kobedde]

Hi Ed, as usual, very well explained.
As you said in the video, I'll play with that a bit to get the hang of it, but your video is a good guide.
Thanks for sharing this.

[boopeshkumarprabhakaran]

Great...I have few of my scrappers running in node red...which gives more flexibility to write manual codes in funtion node.. I mostly scrap from websites which requires logins... Could be great if you do 2nd part of this video.. And include websites which require login authentication

[jaypowa]

Thanks for the video. It's east to understand what you explain.

[RobertWebbe]

I had already come up with such a plan and was very happy when I saw your video. Only my energy supplier (Greenchoice) does not offer a rate sheet via the public internet, I must first log in to the My Greenchoice environment to view the prices of my contract. I've tried it myself with the multiscraper but can't figure it out. Could you also make a video for such situations?

[MichaelXiong-rv]

Nice breakdown of the process! Just curious, how does HasData compare for scraping different types of data?

[sevagjb]

thanks Ed, i need to focus more to learn this, very interesting function to automat

[Timichaud]

cowww!!! I have now so much ideas hahaha thank you sir

[robhuijting]

Oke, briljant tut again, now i have to find the Vattenfall prices haha, Oh and the soccermatches (like from flashscore) :)

[markuss]

it's fantastic, thanks I have scraped the bus time table real time.. but how to access to USERNAME and PASSWORD website? I tried so many time but no result

[ihavn]

Great video.
I'm pretty new to HA, so bare over with my question. How can I turn on the logger to see what is wrong. Thanks

[VWTesla]

I wanted a calendar for USPS Holidays and so I'm scraping a list from a website and using that to flag when it's a USPS Holiday or not. I couldn't find an iCal file that didn't include Valentine's Day which is ordinarily a delivery day so I had to scrape.

[cpreecs]

Thanks for the video, got me going on scraping. But where is the split function shown around 9:00 documented? I'm trying to better understand it so I can use it in other use cases. Thanks!