Skip to main content

​
Overview

The Grafana integration allows you to connect your Grafana instance to incident.io to automatically create incidents from Grafana alerts, enrich alerts with organizational context, and attach dashboard screenshots to incidents for faster debugging.

​
What you can do with Grafana

With Grafana, you can:
  • Create incidents automatically from Grafana alerts by setting up alert sources. Alert sources work without connecting Grafana.
  • Get a screenshot of the alerting panel on each Grafana alert, once Grafana is connected
  • Enrich alerts with attributes using labels on your Grafana alert rules to map to services, teams, and other catalog types
  • Sync telemetry dashboards to your incident.io catalog for easy reference, if your plan includes Investigations

​
Connecting Grafana to incident.io

Prerequisites Before connecting, you’ll need:
  1. Grafana API URL: the hostname of your Grafana instance, such as acme.grafana.net.
  2. Service account token: a non-expiring Grafana service account token. The service account needs either:
    • The basic Viewer role; or
    • The fixed roles Dashboards → Reader, Folders → Reader, and Annotations → Reader.
To create a service account token in Grafana:
  1. Navigate to Administration > Users and access > Service accounts in Grafana
  2. Click Add service account
  3. Give it a descriptive name like “incident.io Integration”
  4. Set it to the Viewer role, or assign the fixed roles above
  5. Click Add service account token and choose No expiration. The connection stops working when a token expires.
Connection steps
  1. In incident.io, go to Settings > Integrations
  2. Find Grafana and click Connect
  3. Enter your Grafana API URL and service account token
  4. Click Connect
Once connected, incident.io will validate the connection by attempting to access your Grafana instance.

​
Setting up Grafana alert sources

Alert sources allow you to automatically create incidents from Grafana alerts. Creating an alert source
  1. Open Alert sources in incident.io
  2. Click Create Alert Source
  3. Select Grafana as the source type
  4. Configure your alert source settings (name, routing rules, etc.)
  5. Copy the webhook URL provided (it will look like https://api.incident.io/v2/alert_events/grafana/{alertSourceId})
Configuring Grafana to send alerts
  1. In Grafana, navigate to Alerting > Contact points
  2. Click Add contact point (or edit an existing one)
  3. Select Webhook as the type
  4. Paste the incident.io webhook URL
  5. Under Authentication, select Bearer Token and paste the token provided by incident.io Note: For older Grafana versions, you may need to use HTTP Basic Auth instead
  6. Save the contact point
Alert grouping options When configuring your alert source, you can choose how Grafana alert groups map to incident.io alerts:
  • Apply Grafana’s grouping (recommended) - Creates a single incident.io alert for each Grafana alert group, keeping related alerts together
  • Ignore Grafana’s grouping - Creates separate incident.io alerts for each alert within a Grafana alert group

​
Enriching alerts with tags

You can tag your Grafana alert rules with labels that map to incident.io catalog types (like services, teams, or custom catalog types), then map those labels to alert attributes in your alert source. Adding tags to Grafana alerts
  1. In Grafana, edit your alert rule
  2. Add labels in the format {catalog-type-name} with the value being the catalog entry name
  3. In your Grafana alert source in incident.io, map the label to an alert attribute of that catalog type
Example: Add a label service with value payment-api, and map the service label to a Service attribute. When this alert fires, incident.io finds the Service entry named payment-api and sets it on the alert’s Service attribute. Common catalog types to use:
  • service - Link alerts to specific services
  • team - Route to the right team
  • Any custom catalog types you’ve created in incident.io
Still got questions? Reach out to support@incident.io or message us in your Slack Channel!