Skip to main content

Context

Datadog sends a webhook every time a monitor changes state: when it triggers, warns, re-notifies and recovers. incident.io uses the deduplication_key in each webhook to decide whether the event belongs to an alert that is already firing. If your deduplication key is not set correctly, events can be merged into an alert you didn’t expect, or a recovery might not resolve the alert it should. Use this deduplication key in your Datadog webhook payload:
This is the key in the payload we give you when you create a Datadog alert source. It gives you one incident.io alert for each time a monitor fires, from the first trigger until it recovers:
  • Every event in one cycle (Warn, Triggered, Re-Triggered, Renotify and Recovered) has the same key, so they all update the same alert, and the Recovered event resolves it.
  • When the monitor fires again after it recovers, Datadog starts a new cycle, so you get a new alert and a new page.
We’d recommend you don’t add $ALERT_TRANSITION, $ALERT_TYPE or $ID to the key. The Recovered event has a different transition and type to the event that fired the alert, so it won’t match the open alert and the alert won’t resolve. $ID changes on every event, so every event creates a new alert.

When a monitor goes from Warn to Alert

A monitor that goes from Warn to Alert in one cycle sends both events with the same deduplication key, so they update one alert. If your alert source sets the priority from the payload (for example from alert_priority or alert_type), we evaluate the priority again on each event. When the priority goes up, your alert route runs again and pages anyone it needs to at the new priority. If you set a fixed priority on your alert source, the priority can’t change between events. To page differently for Warn and Alert, set the priority with an expression. For example, you can use a lower priority when alert_type is warning.