Skip to main content
The catalog importer is our open-source command-line tool for syncing data into your Catalog from anywhere: files in your repositories, Backstage, an internal API, or your data warehouse. Run it on a schedule or from CI, and your catalog types and entries stay in step with the source. Use it when your data doesn’t live in a tool we have a native integration for, or when you want control over exactly which types, entries, and attributes you import. If you’d rather manage a type in a repository without writing the config yourself, the Manage in GitHub button sets up the importer for you. See Managing catalog types in GitHub.

​
How it works

You describe what to import in a config file, written in Jsonnet, YAML, or JSON. The config holds a sync_id and a list of pipelines. Each pipeline has:
  • Sources: where the data comes from, like a file, a command, or the Backstage API.
  • Outputs: the catalog types to create, and how each source record maps to an entry’s name, external ID, and attributes.
Every time you run sync, the importer brings the catalog types in your config into line with whatever the sources return. In practice that means:
  • Entries are created and updated to match the source. Each one is identified by its external_id, so renaming an entry doesn’t create a new one.
  • Entries that have disappeared from the source are deleted. They’re archived rather than gone for good, and one comes back if its external_id reappears. If a source suddenly returns nothing at all, the importer stops rather than emptying the type, unless you’ve passed --allow-delete-all.
  • Only the types the importer owns are touched. Types are tagged with your sync_id, so other importers, and types you manage in the dashboard, are left alone. Do manage each type in one place though: if two pipelines produce entries with the same external_id they’ll overwrite each other on every run, and a type managed with Terraform shouldn’t also be synced by the importer.
  • Types you’ve removed from the config are left where they are, unless you pass --prune.
If a source fails partway through and returns fewer entries than it should, the importer treats the missing ones as deleted. Make sure your exec commands exit with an error when they fail, rather than printing partial output, so the sync stops instead.

​
Taking over a type you built in the dashboard

To start syncing a catalog type you created in the dashboard, give its output the same type_name, and declare every attribute it already has. If the config leaves an existing attribute out, the sync fails. For attributes whose values you still want to edit in the dashboard, like a Team type’s escalation paths, set schema_only: true. See Schema-only attributes.

​
Get started

  1. Install the importer:
    There are also binaries on the releases page, and a Docker image. Stick to the latest release, as recent versions are much faster on large catalogs.
  2. Create an API key with the View catalog and Manage catalog permissions, and export it:
  3. Generate a starting config, choosing a template for Backstage, Jira Service Management Assets, or files:
  4. Check the config, and preview the changes without making them:
    The dry run prints a diff of every change. Entries it would delete are listed under Deleting unmanaged entries, with each value going to nil or empty.
  5. Run the sync:

​
Sources

See Sources for every option.

​
Import from anything with exec

The exec source runs a command and imports whatever JSON or YAML it prints. Use it to load data from somewhere the other sources don’t reach, like an internal API, a query against your data warehouse, or a file that needs reshaping first. This pipeline reshapes a JSON file with jq, and imports each service with its owning team:
The same approach loads a customer list from BigQuery with bq query --format=json, or entries from an internal API with curl. See the exec source for both.
The command runs wherever the importer runs, so the tool it calls needs to be installed there. The Docker image includes only the importer itself, so build your own image on top of it if you need jq, bq, or anything else.

​
Run it on a schedule

The usual setup is to run the importer from CI, so the Catalog updates whenever the source changes, or on a schedule. The importer’s docs have ready-made configs for GitHub Actions, CircleCI, and GitLab CI. Before you use one, update the importer version it installs to the latest release. Run sync with --dry-run on branches, and only run a real sync, or --prune, from your main branch. Pass --source-repo-url with the URL of the repository that holds your config. The Catalog links each type to that repository and stops it being edited in the dashboard, so changes go through the repository instead.

​
Learn more

The catalog importer documentation covers: