Guide to Grafana 101: Getting Started With Alerts

Timescale Developer Advocate, #Grafana enthusiast, and time-series pro @avthars breaks down how alerting works in Grafana (including a handy cheatsheet for Grafana's alert states and alert lifecycles).

You’ll learn how to define and configure 3 kinds of alerts for anomalies in the metrics you care about, trigger notifications via #PagerDuty, Slack, and #OpsGenie, and customize alerts for your projects.

🛠 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝘁 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀

🐯 𝗔𝗯𝗼𝘂𝘁 𝗧𝗶𝗺𝗲𝘀𝗰𝗮𝗹𝗲
At Timescale, we are dedicated to serving developers worldwide, enabling them to build exceptional data-driven products that measure everything that matters. Analyzing this data across the time dimension ("time-series data") enables developers to understand what is happening right now, how that is changing, and why that is changing. We are backed by top-tier investors with a track record of success in the industry.

💻 𝗙𝗶𝗻𝗱 𝗨𝘀 𝗢𝗻𝗹𝗶𝗻𝗲!

📚 𝗖𝗵𝗮𝗽𝘁𝗲𝗿𝘀:
⏱ 0:00 ⇒ Introduction
⏱ 2:42 ⇒ Alerting Principles
⏱ 5:26 ⇒ Alerting in Grafana
⏱ 14:53 ⇒ Let's Code - Implementing 3 alerting rules and 3 notification channels
⏱ 16:40 ⇒ Demo Alert 1: Alert with FOR sent via Slack
⏱ 25:58 ⇒ Lifecycle of Alerts with FOR
⏱ 30:10 ⇒ Demo Alert 2: Alert without FOR sent via PagerDuty
⏱ 35:56 ⇒ Lifecycle of Alerts without FOR
⏱ 39:43 ⇒ Demo Alert 3: Alert with No Data sent via OpsGenie
⏱ 44:45 ⇒ Lifecycle of Alerts with No Data
⏱ 48:07 ⇒ Resources and Next Steps

[TimescaleDB]

Handy Bookmarks of the main parts of the session:
2:42 - Alerting Principles
5:26 - Alerting in Grafana
14:53 - Let's Code - Implementing 3 alerting rules and 3 notification channels
16:40 - Demo Alert 1: Alert with FOR sent via Slack
25:58 - Lifecycle of Alerts with FOR
30:10 - Demo Alert 2: Alert without FOR sent via PagerDuty
35:56 - Lifecycle of Alerts without FOR
39:43 - Demo Alert 3: Alert with No Data sent via OpsGenie
44:45 - Lifecycle of Alerts with No Data
48:07 - Resources and Next Steps

[vanglequy]

Sorry if I missed it somewhere, but it is useful to point out the version of grafana and other things you are demoing. Thanks for very useful tutorials anyway.

[sugardaddy]

Well done! Well explained! The notification channels are well known and common. :)

[wildflowers]

Great video! It's very confusing that the alert frequency definition is above the condition; especially seeing as it's the opposite, spatially, in the alerts not created from graphs in newer versions of grafana.

[dickymr]

How to show the red ticking background when alert is firing? My grafana is 9.1.7 but only show red hearth when firing

[AlgoTribes]

i am trying to include an alert image with the notification message..couldn't do it...i am receiving the alert message but without the image..can someone tell what needs to be done..i am trying to send to the slack using webhook

[socencounter]

Is it possible to configure an alert dashboard that simply lists the alerts as they come and go, without a notification channel, we simply want to manually view all alerts in one dashboard/list which refreshes every 5 minutes?

[cuteangel]

For a circuit breaker alert what value should be put in FOR

[evanlee]

Is it free for creating alerts or paid in grafana

[danielantunes]

Hi, very good session but I had a doubt. Is it possible that a created alarm system only triggers the alarm between two different times? For example, between 09.00am and 23.00pm?

[chetanchetan-dnpr]

on my grafana no alert icon show this is very confusing i have grafana/grafana-enterprise
what is your virsion or it is free ???

[davidj]

You created 1 alert for 1 server and 1 metric. How do you do this in large scale applications of 100 servers? If I want a separate alert for CPU utilization on 100 servers how do I set this up?

[harsha]

Hi Avthar,
If we want to create an alert for the entire dashboard, is there any feasibility for doing it.

[prawininstro]

Great session....while setting up email alert rule, alerting state provide message and metric name. But after it back to OK state the same message again triggering without metrics. How can we solve this.Ok state message bit mis leading🤔

[gautamdikshit]

Hello, I'm using grafana 8 in a GCP VM. I'm monitoring CloudSQL metrics for a number of services which are also running in GCP VMs. My requirement is to monitor 1) Memory Utilization 2) CPU Utilization and 3) Server Uptime and send alerts to the concerned staff if the usage is more than 95%. I'm not using prometheus+cadvisor for this. I'm monitoring it directly from the GCP plugin in grafana.

By default, the unit of y axis is in percentile. If I retain that unit, what number should I have to give in the condition such that I will receive alerts exactly at 95%? Also, even if I change the y axis unit from percentile to percentage and save the alert rule, the state immediately goes from OK to pending even if usage is less than 1%. I receive a lot of alerts unnecessarily. The condition is this - WHEN avg() OF query(A, 5min, now) IS ABOVE 95. What am I doing wrong here? How do I set the condition correctly? Honestly, I don't even understand the difference between percentage and percentile. Please help.

[Alicetian_Deutschland]

thanks for the video, now I understand why i can't create an alert based on table format. so how could we create an alert based on table format?

[pgpvehe]

Thanks for the very informative video @TimescaleDB.
Could you please help clarify this? Is it possible to create an alert for long running queries in Redshift? The notification message should contain the query_id and user_id. It could be either be by setting the query monitoring rule within Redshift and then fetching the log from table or by extracting results from table (both using the query) The goal is to send couple of columns from the result set of the query as the notification message on Slack. Looking forward to your reply. Tried every other possible solution but in vain. Really hoping there's a way around it using Grafana.

[vijaysinghrana]

sir Kindly request please suggest me any workaround to find division or percentage in grafana or panel with elasticsearch please help sir

[vijaysinghrana]

sir please help to find percentage or division in grafana please suggest any solution sir

[NiteshKumar-tcjb]

Is Grafana Alert feature is available with open source licence or it only comes with enterprise licence?