/ #grafana #alerting 

Set Up Grafana Alerts and Notifications via Slack, Email, or Webhook

I’ll admit it: I used to be the person who constantly refreshed my Grafana dashboards, nervously watching a CPU usage graph creep upwards. It was like babysitting a pot of water, waiting for it to boil. This, of course, is not a scalable strategy for managing a smart home system or any observability stack. The whole point of having these beautiful dashboards is to be proactive, not reactive.

That’s where Grafana alerts come in. They are the silent, ever-vigilant guardians of your data. When something goes wrong—a server’s memory is exhausted, your home office temperature sensor stops reporting, or your website’s response time spikes—they can shout about it through a channel you’re actually paying attention to, like Slack or email.

In this guide, I’ll walk you through the process I used to set up Grafana alerts and notifications. We’ll configure the alert channels (Slack, Email, Webhook) first, and then build a practical alert rule to bring it all to life.

What You’ll Need

Before we start, let’s make sure you have everything you need. The good news is, you don’t need much hardware for this one!

  • A Running Grafana Instance: This could be Grafana Cloud (the free tier is plenty for this) or a self-hosted version (I run mine on a Raspberry Pi 4). Version 9.0 or higher is ideal for the latest alerting features.
  • Administrator Access: To configure notification channels, you’ll need to have the Admin role for your Grafana organization, or have the permissions delegated to you.
  • A Slack Workspace (Optional): If you want Slack notifications, you’ll need a workspace where you can create an incoming webhook.
  • An SMTP Server (Optional): For email alerts. This could be Gmail’s SMTP, your ISP’s server, or a service like SendGrid.
  • A Webhook URL (Optional): For connecting to other services like Discord, PagerDuty, or a custom API.

Heads up! Grafana completely overhauled its alerting system in versions 8 and 9, moving away from the legacy "Alerting" tab to a unified "Alerting" experience. This tutorial is for the new, unified alerting system. If your menus look different, you might be on an older version.

Step 1: Configuring Your Notification Channels

Think of notification channels as the pipes that carry your alerts from Grafana to the outside world. We need to lay these pipes before we can turn on the water. We’ll do this in the Alerting section of Grafana.

Navigate to Alerting > Contact points. This is where we define the destinations for our alerts.

Option A: Setting Up Slack Notifications

Slack is my go-to for immediate, high-visibility alerts.

  1. Create a Slack App: Go to api.slack.com/apps and click “Create New App.” Choose “From scratch,” give it a name (e.g., “Grafana Alerts”), and select your workspace.
  2. Activate Incoming Webhooks: In your app’s settings, navigate to “Incoming Webhooks” and toggle the switch to “On.”
  3. Add a New Webhook: Scroll down and click “Add New Webhook to Workspace.” Choose the channel you want the alerts to post to (I created a dedicated #grafana-alerts channel) and allow access.
  4. Copy the Webhook URL: You’ll get a unique Webhook URL. It will look something like https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZZZZZZZ. Copy this; it’s your secret key.
  5. Configure in Grafana: Back in Grafana’s “Contact points,” click “Add contact point.”
    • Name: Slack Alerts
    • Contact point type: Select Slack
    • Webhook URL: Paste the URL you copied from Slack.
    • Leave the other settings as default for now.

Click “Test” to send a trial notification to your Slack channel. If you see a message from Grafana, you’re golden!

Option B: Setting Up Email Notifications

For less critical alerts or daily digests, email is a classic and reliable choice. First, we need to configure Grafana’s SMTP settings.

  1. Configure SMTP in grafana.ini: If you’re self-hosting Grafana, you’ll need to edit your grafana.ini configuration file. If you’re using Grafana Cloud, you can skip to step 2, as these settings are often available in the UI.

    Locate the [smtp] section in your grafana.ini file and modify it. Here’s an example for Gmail:

    [smtp]
enabled = true
host = smtp.gmail.com:587
user = your.email@gmail.com
# If you use 2FA, you need an App Password, not your main password.
password = your_app_specific_password
from_address = your.email@gmail.com
from_name = Grafana Alerts

    Restart your Grafana server for these changes to take effect.

  2. Create the Email Contact Point: In the Grafana UI, go back to Alerting > Contact points and click “Add contact point.”

    • Name: Email Alerts
    • Contact point type: Select Email
    • Addresses: Enter the email addresses (one per line) that should receive the alerts.
    • Single email: Toggle this on if you want all addresses to be in a single email (BCC). Toggle it off to send individual emails.

Click “Test” to receive a test email. Check your spam folder if it doesn’t arrive!

Option C: Setting Up a Generic Webhook

Webhooks are the most flexible option, allowing you to connect to almost any service.

  1. Get your Webhook URL: This depends on the service. For Discord, it would be a “Webhook” URL from a channel’s settings. For a custom script, it would be the URL where your script is listening.
  2. Configure in Grafana: In “Contact points,” add a new one.
    • Name: My Webhook
    • Contact point type: Select Webhook
    • URL: Paste your webhook URL here.
    • You can often leave the other fields as default, but some services may require specific HTTP Methods (POST) or additional headers.

Step 2: Creating Your First Alert Rule

Now for the fun part: telling Grafana what to alert on. We’ll create a simple but practical alert rule. Let’s imagine we want an alert if a server’s available memory drops below 10%.

Navigate to Alerting > Alert rules and click “Create alert rule.”

1. Define the Rule

  • Rule name: Low Memory Warning
  • Folder: Choose an existing folder or create a new one to keep your alerts organized.

2. Set the Query and Condition

This is the brain of your alert.

  1. In the “Query A” section, select your data source (e.g., Prometheus).
  2. Write a query that fetches the metric you want to alert on. For available memory percentage, a PromQL query might look like this:
    (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
  3. Under “Alert condition,” you tell Grafana how to evaluate the query. Set it to B (the letter of your query, likely A), and then define the condition. We want to alert when the value is above 90 (since we’re calculating used memory percentage).

3. Configure the Alert Evaluation Behavior

This tells Grafana how often to check the condition.

  • Evaluate every: 1m (Check the rule every minute)
  • For: 2m (The condition must be true for 2 consecutive minutes before firing. This prevents flapping alerts from brief spikes.)

4. Add Details and Notifications

This is where we connect the alert to the channels we built earlier.

  • Summary: A short, human-readable summary. Memory usage is high on {{ $labels.instance }}.

  • Description: A more detailed message. You can use Go templating here to include dynamic data from the alert!

    🚨 **Low Memory Alert**
Instance: `{{ $labels.instance }}`
Current Value: `{{ printf "%.2f" $value }}%`
The available memory is critically low. Please investigate.

    I learned the hard way that a good description saves you a lot of time when you’re woken up at 3 AM. You can find more details on custom messages in my dedicated post on Grafana alert custom messages.

  • Contact points: In the “Notify” section, click “Add contact point” and select the Slack Alerts and/or Email Alerts contact points we created.

5. Save and Test

Click “Save rule” at the bottom. Your alert is now active! To test it, you could temporarily lower the threshold to a value you know is currently being exceeded. You should see the alert fire and a message appear in your chosen channels.

Pro Tip: Use labels! When you create an alert rule, you can add custom labels (e.g., `severity=warning`, `team=infra`). You can then use these labels in "Notification policies" to route different types of alerts to different channels. For example, send all `severity=critical` alerts to Slack and PagerDuty, and `severity=warning` alerts only to email.

Troubleshooting Common Issues

  • “Test” works, but real alerts don’t arrive: Double-check your alert rule’s condition and evaluation interval. The “Test” button bypasses the condition logic, so the rule itself might not be firing.
  • Slack message is poorly formatted: You can use Slack’s Block Kit Builder to design a custom JSON payload for your webhook, making the alerts much prettier and more actionable.
  • Emails going to spam: Ensure your from_address is valid and that your SMTP server is properly configured. The “Test” email from Grafana is a good indicator of whether your server’s reputation is an issue.
  • Alert is in a pending state: This is normal! It means the condition is met, but the “For” duration hasn’t elapsed yet. Grafana is waiting to see if the problem is persistent.

Where to Go From Here

Once you’ve mastered a basic alert, the possibilities are endless. You can create alerts for:

  • Smart Home: Alert when your basement temperature drops below freezing, or when a Zigbee device becomes unavailable for more than 10 minutes.
  • Website Uptime: Use Grafana’s synthetic monitoring to alert if your homepage returns anything other than a 200 status code.
  • Business Metrics: Alert when the number of user sign-ups per hour falls below a certain threshold.

The key is to start simple, get one alert working perfectly, and then expand. Soon, you’ll have a fully automated watchdog for your entire system, and you can finally stop watching that pot of water.

Frequently Asked Questions (FAQ)

Q: Can I send a single alert to multiple channels? A: Absolutely! In your alert rule, you can add multiple contact points in the “Notify” section. You can also use Notification Policies to route alerts based on labels.

Q: What’s the difference between Grafana alerts and Prometheus alerts? A: Prometheus Alertmanager is a separate, powerful system for managing alerts from Prometheus. Grafana alerts are built into Grafana and can work with many data sources (not just Prometheus). For simpler setups, Grafana alerts are often easier to manage.

Q: How do I silence an alert for maintenance? A: In the Alerting > Silences section, you can create a silence for a specific alert rule or set of labels. This is perfect for planned downtime.

Q: My alert is firing, but the graph doesn’t show a problem. Why? A: Check your query and condition. A common culprit is the “Reduce” function. If you’re using last(), make sure it’s being applied to the correct time range relative to your evaluation interval.