> ## Documentation Index
> Fetch the complete documentation index at: https://docs.incident.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Sync catalog data with the catalog importer

> Keep your Catalog in step with the systems that already know about your services, teams, and customers.

The [catalog importer](https://github.com/incident-io/catalog-importer) is our open-source command-line tool for syncing data into your [Catalog](/catalog/catalog-setup) from anywhere: files in your repositories, Backstage, an internal API, or your data warehouse. Run it on a schedule or from CI, and your catalog types and entries stay in step with the source.

Use it when your data doesn't live in a tool we have a [native integration](/catalog/importing-data) for, or when you want control over exactly which types, entries, and attributes you import. If you'd rather manage a type in a repository without writing the config yourself, the **Manage in GitHub** button sets up the importer for you. See [Managing catalog types in GitHub](/catalog/github-managed-types).

## How it works

You describe what to import in a config file, written in Jsonnet, YAML, or JSON. The config holds a `sync_id` and a list of **pipelines**. Each pipeline has:

* **Sources**: where the data comes from, like a file, a command, or the Backstage API.
* **Outputs**: the catalog types to create, and how each source record maps to an entry's name, external ID, and attributes.

Each time you run `sync`, the importer makes the catalog types in your config match the source:

* **It creates and updates entries** to match what the sources return, identifying each one by its `external_id`, so renaming an entry doesn't create a new one.
* **It deletes entries that are no longer in the source.** Deleted entries are archived, and one comes back if its `external_id` reappears in the source. If a source suddenly returns nothing, the importer stops rather than empty the type, unless you pass `--allow-delete-all`.
* **It only touches the types it owns.** Types are tagged with your `sync_id`, so separate importers, and types you manage in the dashboard, are left alone. Manage each type in one place: if two pipelines produce entries with the same `external_id`, they overwrite each other on every run, and a type managed with [Terraform](/admin/terraform) shouldn't also be synced by the importer.
* **It leaves types you've removed from the config**, unless you pass `--prune`.

<Warning>
  A source that fails partway through can return fewer entries than it should, and the importer deletes the rest. Make
  sure your `exec` commands exit with an error when they fail, rather than printing partial output, so the sync stops
  instead.
</Warning>

### Taking over a type you built in the dashboard

To start syncing a catalog type you created in the dashboard, give its output the same `type_name`, and declare every attribute it already has. If the config leaves an existing attribute out, the sync fails. For attributes whose values you still want to edit in the dashboard, like a Team type's escalation paths, set `schema_only: true`. See [Schema-only attributes](https://github.com/incident-io/catalog-importer/blob/master/docs/outputs.md#schema-only-attributes).

## Get started

1. Install the importer:

   ```bash theme={null}
   brew tap incident-io/homebrew-taps
   brew install catalog-importer
   ```

   There are also binaries on the [releases page](https://github.com/incident-io/catalog-importer/releases), and a [Docker image](https://hub.docker.com/r/incidentio/catalog-importer). Use the latest release: recent versions are much faster on large catalogs.

2. Create an [API key](/admin/api-keys) with the **View catalog** and **Manage catalog** permissions, and export it:

   ```bash theme={null}
   export INCIDENT_API_KEY=<YOUR_API_KEY>
   ```

3. Generate a starting config, choosing a template for Backstage, Jira Service Management Assets, or files:

   ```bash theme={null}
   catalog-importer init
   ```

4. Check the config, and preview the changes without making them:

   ```bash theme={null}
   catalog-importer validate --config importer.jsonnet
   catalog-importer sync --config importer.jsonnet --dry-run
   ```

   The dry run prints a diff of every change. Entries it would delete are listed under **Deleting unmanaged entries**, with each value going to `nil` or empty.

5. Run the sync:

   ```bash theme={null}
   catalog-importer sync --config importer.jsonnet
   ```

## Sources

| Source | Loads data from |
| - | - |
| `inline` | Entries written directly in the config |
| `local` | JSON, YAML, or Jsonnet files next to the config |
| `github` | Files across your GitHub repositories |
| `backstage` | The Backstage API. See [Backstage](/catalog/backstage). |
| `graphql` | A GraphQL API |
| `exec` | The output of any command |

See [Sources](https://github.com/incident-io/catalog-importer/blob/master/docs/sources.md) for every option.

### Import from anything with `exec`

The `exec` source runs a command and imports whatever JSON or YAML it prints. Use it to load data from somewhere the other sources don't reach, like an internal API, a query against your data warehouse, or a file that needs reshaping first.

This pipeline reshapes a JSON file with `jq`, and imports each service with its owning team:

```jsonnet theme={null}
{
  sync_id: 'my-org/catalog',
  pipelines: [
    {
      sources: [
        { exec: { command: ['jq', 'to_entries | map(.value)', 'services.json'] } },
      ],
      outputs: [
        {
          name: 'Service',
          description: 'Services, synced from services.json.',
          type_name: 'Custom["Service"]',
          source: { name: '$.name', external_id: '$.id' },
          attributes: [
            { id: 'team', name: 'Team', type: 'String', source: '$.team' },
          ],
        },
      ],
    },
  ],
}
```

The same approach loads a customer list from BigQuery with `bq query --format=json`, or entries from an internal API with `curl`. See [the `exec` source](https://github.com/incident-io/catalog-importer/blob/master/docs/sources.md#exec) for both.

<Note>
  The command runs wherever the importer runs, so the tool it calls needs to be installed there. The Docker image
  includes only the importer itself, so build your own image on top of it if you need `jq`, `bq`, or anything else.
</Note>

## Run it on a schedule

Run the importer from CI, so the Catalog updates whenever the source changes, or on a schedule. The importer's docs have ready-made configs for [GitHub Actions, CircleCI, and GitLab CI](https://github.com/incident-io/catalog-importer/blob/master/docs/deploying.md). Before you use one, update the importer version it installs to the latest release. Run `sync` with `--dry-run` on branches, and only run a real sync, or `--prune`, from your main branch.

Pass `--source-repo-url` with the URL of the repository that holds your config. The Catalog links each type to that repository and stops it being edited in the dashboard, so changes go through the repository instead.

## Learn more

The [catalog importer documentation](https://github.com/incident-io/catalog-importer/tree/master/docs) covers:

* [Configuration](https://github.com/incident-io/catalog-importer/blob/master/docs/config.md), including multiple pipelines and filtering
* [Outputs](https://github.com/incident-io/catalog-importer/blob/master/docs/outputs.md), including attributes that reference other catalog types, enums, and team ownership
* [Expressions](https://github.com/incident-io/catalog-importer/blob/master/docs/expressions.md) for mapping and filtering source data
* [Troubleshooting](https://github.com/incident-io/catalog-importer/blob/master/docs/troubleshooting.md)