> ## Documentation Index
> Fetch the complete documentation index at: https://docs.incident.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Adding Icinga 2 as an alert source

> Turn Icinga 2 host and service notifications into pages and incidents, and close them out on recovery.

Icinga 2 monitors your hosts and services and decides when something is worth telling a human about. Connect it to incident.io to page the right people when a check goes critical, and to resolve the alert automatically when that check recovers.

Icinga 2 sends alerts to incident.io through the [icinga2-plugin](https://github.com/incident-io/icinga2-plugin) integration, pointed at an [HTTP alert source](/alerts/custom-http-sources) you create in incident.io. The integration is a notification command: Icinga runs it when a notification fires, and it posts an alert carrying the host, service, state, and any custom variables you choose to send. Once connected, Icinga alerts flow into your alert routes for escalation and incident creation, just like any other source.

Two things follow from Icinga owning the decision to notify:

* **Icinga's suppression applies first.** Downtimes, acknowledgements, notification periods, and `times` windows are all evaluated before incident.io sees anything. An object in a scheduled downtime does not page anyone.
* **Nothing changes until you opt in.** Installing the integration sends no alerts. You choose which hosts and services participate with `vars.incident_io = true`.

## Before you start

|                |                                                             |
| -------------- | ----------------------------------------------------------- |
| Icinga 2       | 2.11 or newer                                               |
| Install on     | every master in the master zone, and nothing else           |
| On each master | POSIX `/bin/sh`, `curl`, `sed`, `awk`, `tr`, `od`, `mktemp` |
| Network        | outbound HTTPS from each master to `api.incident.io`        |

Notifications only ever fire from the master zone, so satellites and agents need nothing installed and need no egress. Icinga's config sync distributes `.conf` files but not executables, so each master needs the package.

## Instructions

<Steps>
  <Step title="Create the alert source">
    1. Head over to the [Alerts](https://app.incident.io/~/alerts/sources) section in your incident.io dashboard.

    2. Select the sources tab at the top of the page.

    3. Press the **New alert source** button.

    4. Search for **HTTP**, then click continue to create the alert source. Name it something you will recognize, such as `Icinga 2`.

    5. Leave the source type as **Default**. The integration already sends incident.io's standard alert schema, so you do not need a transform expression.

    6. Copy the **URL** and **bearer token** shown on the setup page. You will paste both into Icinga in step 3.
  </Step>

  <Step title="Install the integration on each master">
    Download the `.deb` or `.rpm` from the [releases page](https://github.com/incident-io/icinga2-plugin/releases) and install it on every master in your master zone:

    ```bash theme={null}
    # Debian and Ubuntu
    sudo apt install ./icinga2-incident-io_0.1.0_all.deb

    # RHEL, Rocky, and SLES
    sudo rpm -i icinga2-incident-io-0.1.0-1.noarch.rpm
    ```

    This installs the notification handler to `/usr/bin/incident-io-icinga` and its Icinga configuration to `/etc/icinga2/conf.d/`. Use whatever configuration management already installs packages on your masters, so a new master does not come up missing it.
  </Step>

  <Step title="Add your credentials">
    On each master, create the credentials file from the example the package ships:

    ```bash theme={null}
    sudo cp /etc/icinga2/conf.d/incident-io-secrets.conf.example \
            /etc/icinga2/conf.d/incident-io-secrets.conf
    sudo chown root:icinga /etc/icinga2/conf.d/incident-io-secrets.conf
    sudo chmod 0640        /etc/icinga2/conf.d/incident-io-secrets.conf
    sudoedit /etc/icinga2/conf.d/incident-io-secrets.conf
    ```

    Fill in the URL and token from step 1, and the base URL of your Icinga Web 2 install:

    ```c theme={null}
    const IncidentIoUrl          = "https://api.incident.io/v2/alert_events/http/SOURCE_ID"
    const IncidentIoToken        = "TOKEN"
    const IncidentIoIcingaWebUrl = "https://icinga.example.com/icingaweb2"

    globals.IncidentIoIcingaWebStyle = "icingadb"
    ```

    `IncidentIoIcingaWebUrl` puts a link back to the object on every alert, so a responder can go from the page to the check in one click. `IncidentIoIcingaWebStyle` decides which Icinga Web 2 front end that link points at: use `icingadb` for Icinga DB Web, or `monitoring` for the older monitoring module. If you are unsure, open a host in Icinga Web 2 and look at the path in your browser.

    <Warning>
      Keep this file out of `zones.d`. Config sync would stage a copy of your token under `/var/lib/icinga2/api/zones/` on every node in the zone.
    </Warning>
  </Step>

  <Step title="Choose which objects alert">
    Mark the hosts and services you want paged on. A host set to `true` covers its services too:

    ```c theme={null}
    object Host "web-01.dc-fra" {
      import "generic-host"
      address = "10.0.1.4"

      vars.incident_io = true
    }
    ```

    Set `vars.incident_io = false` on an individual service to carve it out again. To route your whole estate, change the `assign where` clauses in `/etc/icinga2/conf.d/incident-io-notifications.conf` to `assign where true`.
  </Step>

  <Step title="Validate and reload">
    ```bash theme={null}
    sudo icinga2 daemon -C && sudo systemctl reload icinga2
    ```

    If validation complains that `IncidentIoUrl` is undefined, the credentials file from step 3 has not been created yet.
  </Step>
</Steps>

## Confirm it works

Build a payload without sending anything, to check the handler runs:

```bash theme={null}
INCIDENT_IO_URL=x INCIDENT_IO_TOKEN=x \
HOST_NAME=web-01 SERVICE_NAME='disk /' STATE=CRITICAL \
NOTIFICATION_TYPE=PROBLEM OUTPUT='DISK CRITICAL - 12% free' \
  incident-io-icinga --dry-run
```

Then send a real one. In Icinga Web 2, open any object you opted in and use **Send custom notification**. The alert appears in incident.io within a few seconds.

Handler output goes to syslog under the tag `incident-io-icinga`, so `journalctl -t incident-io-icinga` tells you what happened to a notification that did not arrive.

<Tip>
  We recommend opting in a handful of objects, or a non-production zone, and watching those alerts arrive before you enable Icinga alerting across your estate.
</Tip>

## Route alerts with Icinga metadata

Every alert carries a `metadata` object, which you can use to build [alert attributes](/alerts/attributes-and-priorities) and route alerts to the right team. You get these without configuring anything:

| Key                 | Example             | What it is                           |
| ------------------- | ------------------- | ------------------------------------ |
| `host`              | `web-01.dc-fra`     | the Icinga host object               |
| `service`           | `disk /`            | the service, empty for host alerts   |
| `state`             | `CRITICAL`          | the current state                    |
| `notification_type` | `PROBLEM`           | the Icinga notification type         |
| `check_source`      | `satellite-fra-1`   | the node that actually ran the check |
| `hostgroups`        | `["linux", "prod"]` | the host's groups                    |
| `servicegroups`     | `["disk"]`          | the service's groups                 |

You can add anything else Icinga knows. Name existing custom variables to promote them, or set an explicit dictionary:

```c theme={null}
object Host "web-01.dc-fra" {
  vars.incident_io = true

  vars.team     = "payments"
  vars.tier     = 1
  vars.runbook  = "https://wiki.example.com/runbooks/web"

  vars.incident_io_metadata_vars = [ "team", "tier", "runbook" ]
}
```

Promoting variables by name is deliberate rather than exporting everything, because custom variables often hold check credentials. Nothing leaves your estate unless you name it. See [metadata](https://github.com/incident-io/icinga2-plugin/blob/main/docs/METADATA.md) for nested objects, service-level metadata, and estate-wide defaults.

`check_source` is worth calling out for distributed setups. It names the satellite that executed the check, so where your satellites map to datacenters, you can route EU checks to EU on-call.

Once alerts are arriving, [create an alert route](/alerts/getting-started) to filter, group, and escalate them, or to automatically create incidents.

## FAQs

<AccordionGroup>
  <Accordion title="Do alerts resolve on their own?">
    Yes. Icinga sends a `RECOVERY` notification when a check comes good, and the integration resolves the alert it opened. This works because the deduplication key is built only from the host and service names, never from the state, timestamp, or the node that sent it.
  </Accordion>

  <Accordion title="Will two HA masters send me duplicate alerts?">
    No. Icinga's notification component runs with `enable_ha = true` by default, so exactly one master owns notifications at a time and failover is automatic. Even if that assumption broke, both masters would produce the same deduplication key, so incident.io would collapse them into a single alert.
  </Accordion>

  <Accordion title="What happens during a scheduled downtime?">
    Icinga suppresses notifications for objects in downtime, so nothing is sent. If a downtime starts while an alert is already open, incident.io resolves it, so planned maintenance does not leave an alert hanging. The alert fires again if the problem is still there when the downtime ends.
  </Accordion>

  <Accordion title="Do I need to install anything on satellites or agents?">
    No. Notifications only fire from the master zone. Satellites and agents keep running checks and need no egress to incident.io.
  </Accordion>

  <Accordion title="The link on my alerts leads to a 404">
    `IncidentIoIcingaWebStyle` is set to the wrong front end. Icinga DB Web serves objects under `/icingadb/`, and the older monitoring module under `/monitoring/`. Open a host in Icinga Web 2, check which path your browser shows, and set the matching value.
  </Accordion>

  <Accordion title="We manage Icinga with Director. Does that work?">
    The repository ships a Director configuration basket under `contrib/`, which creates the Icinga objects through the Director UI. Director cannot install the handler itself, so you still need the package on every master.
  </Accordion>

  <Accordion title="How is this different from the Nagios integration?">
    Use this one for Icinga 2, and the [Nagios integration](/alerts/nagios) for Nagios Core. Icinga 2 has its own configuration language and notification model, so it gets a notification command that understands host and service objects, custom variables, and satellites, rather than a Nagios plugin.
  </Accordion>
</AccordionGroup>