Skip to main content

Escalation path branches

You might have people who are on-call just during business hours, or where you only want to wake people up when the priority is the highest for an alert. You can now set up rules around priority and/or working hours to ensure you page the right people at the right time without causing additional noise.
The instructions below help you create multiple escalation paths with rules, but you can still create simple escalation paths with the feature as in this help article.

How does it work?

You can start creating a new escalation path from scratch or edit your current paths. With this feature, you can
  • Set up working hours for all levels in the escalation path
  • Configure priorities in incident.io and connect them to your alert source
  • Configure notifications for your devices for high and low-urgency
  • Create branches based on priority and/or working hours
  • Choose which levels should send high or low-urgency notifications
  • Choose to escalate either within a time window or based on your working hours

1. Set up working hours for all levels in the escalation path

You can define one or more named sets of working hours for your escalation path — for example, separate configs for your UK and US teams, each with their own days, times, and timezone. These are then referenced in branches (to route escalations differently based on the time of day) and in ack deadlines (to wait until a team’s working hours begin or end before escalating further).

2. Configure priorities and connect them to your alert source

To use priorities in the escalation path branches, you can create them in Alert Attribute settings. Priorities are now a first-class citizen in incident.io, which means that you wouldn’t need to create those as a custom field in the catalog just to connect with your alert source.
You can create priorities locally just in incident.io or bringing your priorities from your Alert source. We’ll show both cases below.

Create your priorities in incident.io

  1. Head to Alert Attribute settings
  2. Create Priorities, organize them with drag and drop and choose a default value
  3. After creating the priorities, head to your Alert sources to set up the default priority per source.

Bring your priorities from your alert source i.e. Datadog

  1. Go to your Alert sources and choose an alert source
  2. Go to Configure tab and scroll down to Priorities
  3. Choose to use a variable and Add new Expression
  4. Choose query
  5. Choose the payload and Parse
  6. Write to field $.metadata.priority (Example, can be a different payload variable name)
  7. Choose Alert priority from the drop-down
  8. Choose the default if the query doesn’t return the Alert priority
  9. Save the setting by clicking ‘Add’
  10. Save the alert source
You will also be able to see your priorities in Catalog if created or brought through your Alert source.

3. Configure notifications for your devices for high and low-urgency

If you want to enable your workflows to use different notifications depending on priority or working hour branches, you can head to your Preferences and choose the notifications you’d like to receive in both cases.

4. Create branches based on priority and/or working hours

Now that you have all configurations created, you can start your path with or without conditions. Conditions we offer at the moment are working hours and priority. You can choose to start from either You can also configure a branch to take no action, which is useful for intentionally suppressing escalation under certain conditions — for example, silencing low-priority alerts outside working hours. If you want to use both priority and working hours as rules, you should create three branches, like below. You can choose to start your escalations either with working hours or priorities. The example below will do the next
  • 1st branch will send Alerts within working hours with high urgency notifications to the person on call from the Payments team and wait 10 minutes until escalating to the next level
  • Second branch
    • If P1, will send Alerts outside of working hours with high urgency notifications to the person on call from the Payments team and wait 5 minutes before escalating to the next level
    • If other than P1, will send Alerts outside working hours with low urgency notifications to the person on call from the Payments team and wait until working hours begin for that person to acknowledge, before escalating to the next level

5. Choose which levels should send high or low-urgency notifications

Earlier in this article, we showed how every person can set up their preferences on high and low-urgency alerts. To set them up in the Escalation path, just choose either High or Low in every level to choose how you’d want to notify the users.

6. Choose to escalate either within a time window or based on your working hours

If you have used Working hours as a condition in your branch, you’ll receive a few new options in escalation delays: Working hours begin and working hours end. This means that people who are on call will receive the notifications at that moment, but we’ll wait until that time before we escalate to the next level

Escalate to a channel

Choose when an alert should be escalated to a Slack or Microsoft Teams channel of your choice where anyone can acknowledge and start resolving the alert.
  1. Head to Escalation paths
  2. Click ‘Level’ to choose a channel
  3. Search for the channel — both public and private channels are supported
  4. Save your escalation path
Example use cases
  • You can send all escalations to a shared Slack channel to let a group of people know that something is wrong, but still escalate to an actual responder at the same time by selecting “Don’t wait” as a time to acknowledge option.
  • You can set up a branch to deviate low urgency escalations to a Slack channel.

Good to know

  • If you have an escalation path that first pages a slack channel, then the person being on call in the next level will be shown as ‘Level 1’ in your On-call status
  • If you have an escalation path that first pages someone else, then a slack channel, then you, you’ll be shown as ‘Level 2’ in your On-call status

Page a schedule

When you page a schedule from an escalation level, you can choose who to notify. After first selecting a schedule, the dropdown gives you three schedule-wide options, plus a sub-menu for narrowing the level to a specific rotation:
  • Currently on-call: page the person on call across the schedule right now
  • Next on-call: page the next person scheduled to come on call
  • Everyone on schedule: page everyone on call at the same time
  • For a specific rotation: narrow any of the above to a single rotation within the schedule

Currently on-call

The default when you select a schedule. Pages whoever is on call right now, combining all rotations in the schedule.

Next on-call

Pages whoever is scheduled to come on call next. This is most useful as a fallback level. Pair it with Currently on-call on an earlier level so that if the current on-call doesn’t acknowledge, the escalation reaches a different person. Next on-call always finds a real person to page:
  • If someone is currently on call, it pages whoever is due to take over from them, skipping the current on-call user, even if their next shift is back-to-back.
  • If nobody is currently on call (for example, outside working hours), it pages the next person coming on, even if that isn’t until later in the week.

Everyone on schedule

Page everyone at the same time in a schedule. When choosing this, everyone in the same level will be paged at the same time. By choosing ‘Everyone’ it means that we will page everyone in that level. If a person from this level acks, we will not escalate to the next level. By default, we will continue to notify all people in the level until their notification preferences end, though you can configure the path to cancel notifications for others once someone acknowledges. If the level is set to notify more than once, the first acknowledgement also stops any further attempts, for everyone on the level. Read more about notification preferences here.

For a specific rotation

Choosing this opens a sub-menu listing each rotation in the schedule. Pick a rotation, then choose Currently on-call, Next on-call, or Everyone to apply that routing option to just that rotation. This is most useful when a schedule has multiple rotations (for example L1, L2, L3) and you want different escalation levels to target different rotations. You can also target all users on a specific rotation within a schedule, rather than everyone across all rotations. Use this with round robin or ‘all at once’ in the same way as schedule-wide paging.

How to set it up

  1. Head to Escalation paths
  2. Open a level and choose a schedule
  3. Pick a routing option from the dropdown: Currently on-call, Next on-call, Everyone on schedule, or For a specific rotation
  4. If you picked For a specific rotation, choose the rotation, then choose the routing option for that rotation
  5. Save the path

Good to know

  • ‘All at once’ setting will only page those people in a schedule who are on call
  • For cycling through responders within a level, see Round Robin in Escalation paths

Retry a level before moving on

A level notifies its responders once, then moves on to the next level if nobody acknowledges in time. You can instead set a level to notify the same people several times before it moves on, so you give the person on call more than one chance to respond before you widen the page to a bigger group. Open a level in the escalation path editor and click Notify once. Choose More than once, then set how often to notify and how many attempts to make. The level then reads, for example, Notify every 5m, up to 3 times. The first notification counts as attempt one, so three attempts means three notifications in total. Each attempt re-runs every responder’s notification rules from the start, so a responder who is set up to receive a Slack message and then a phone call gets that whole sequence again on each attempt. Retries stop as soon as somebody acknowledges. On a level where everyone must acknowledge, the level still waits for the remaining people until it expires, but nobody is notified again.
The level’s time to acknowledge still decides when the escalation moves on, and it is measured from when the escalation arrived at the level. Retrying does not extend it. This has two consequences:
  • A level can move on before it has made all of its attempts. The editor warns you when this will happen. To fit all of the attempts in, give the level a longer time to acknowledge, or notify less often.
  • A level that has used all of its attempts does not move on early. It waits for the rest of its time to acknowledge, then escalates.
  • You can make between 2 and 10 attempts.
  • The interval between attempts is a whole number of minutes, and at least 1 minute.
  • A level that uses round robin cannot retry, because round robin already works through the level’s responders on a timer. Choose one or the other for a given level.
  • A level that notifies a Slack or Microsoft Teams channel cannot retry.
Each attempt starts your notification rules again from the beginning, so a rule with a delay longer than the interval between attempts only completes on the final attempt. If a level notifies every 5 minutes and your rules call you after 7 minutes, the call only happens on the last attempt. Read more about notification rules here.

Retry across the path

A Retry node sends an unacknowledged escalation back through the path from an earlier point, so a whole sequence of levels runs again. Add it after your last level, choose which node to restart from, and set how many times to repeat. This is a different scope to a per-level retry. A per-level retry notifies one level again and never leaves that level. A Retry node runs the path again and re-evaluates the conditions on the way, so a branch on working hours or priority can send the escalation down a different route the second time. Retries across the path stop when somebody acknowledges, or when the repeats run out.
There is a third kind of retry, which continues to page while the alert is still firing, after an escalation has already been acknowledged. See Repeating acknowledged escalations.

Delay escalation

Add a delay node to pause an escalation before it proceeds to the next level. This is useful for alerts that may resolve on their own, or for holding off on paging someone until working hours. Add a delay node in the escalation path editor just like any other step. Choose your delay mode and save the path. When an alert triggers the escalation, it pauses at the delay node for the configured period. If the alert resolves while waiting, the escalation stops entirely. Delay nodes have two modes:
  • Delay for a fixed duration: Pause for a set number of minutes (e.g., 5, 10, or 15 minutes) before continuing. If the alert resolves during the delay, nobody gets paged.
  • Delay until working hours: Hold the escalation until the configured working hours begin, so responders aren’t woken up overnight for non-critical alerts.

Restoring a previous version

If you make a change to an escalation path and need to undo it, you can preview and restore previous versions from the escalation path’s dropdown menu.

Read more about Escalation paths

Dynamically setting an escalation path Escalating to the right team from an alert Round Robin in Escalation paths
If you have any feedback or feature requests on the feature, please send us a message to support@incident.io - we’d love to hear from you!