Skip to main content
Elasticsearch stores and searches your logs. Investigations query the index patterns you point them at to read what your services recorded around the time of an incident — the errors, the warnings, the request that failed.
You connect Elasticsearch directly, with your cluster’s endpoint and credentials. It works the same whether you run Elastic Cloud or manage your own cluster.

​
What we support

Investigations query Elasticsearch with its Query DSL against the log index patterns you configure, in two ways:
  • Log searches — pull back the actual documents: what a service logged at the time, whether errors appeared on an endpoint, or whether a message started showing up right after a deploy.
  • Aggregations over logs — turn matching documents into a time series, so investigations can graph trends straight from your logs: an error count climbing, request volume dropping away, the frequency of a particular message across the incident window. You get a chart of what your logs were doing even where you never set up a dedicated metric for it.

​
Querying your index patterns

Elasticsearch doesn’t expose one set of logs — it holds whatever indices and data streams you ship to it, and the useful ones differ by team. So you tell investigations which index patterns to query, for example logs-* for application logs or a security audit stream, and each pattern becomes its own queryable source you can enable or disable independently. Until you configure at least one index pattern, an Elasticsearch connection has nothing to query. Add your patterns during setup, then enable the ones your responders reach for.

​
Learning your fields and mappings

A log document in Elasticsearch can carry hundreds of fields, and a query is only as good as knowing which ones exist and what they hold. For each index pattern, investigations read its mapping to learn the fields and their types, then sample recent documents to learn which fields are actually populated, how many distinct values each holds, and the common values for the ones worth filtering on — a status code, a service name, a customer. They also learn the timestamp field your index uses and how long it retains data, so queries scope to the incident window and don’t reach past what’s still there. This means investigations can filter on the fields that matter in your logs without you describing your schema by hand. How that learning works is covered in How telemetry works.

​
Connecting Elasticsearch

Connect Elasticsearch from the Investigations settings by adding a telemetry data source and choosing Elasticsearch. What you’ll need:
  • Your cluster’s endpoint. For Elastic Cloud, this is your deployment’s Cloud ID. For a self-managed cluster, this is one or more node addresses.
  • Credentials. Either an API key or a username and password.
  • A CA certificate (self-managed only, optional) — provide one if your cluster uses a self-signed or internal certificate authority. Elastic Cloud uses public certificates, so this isn’t needed there.
To connect:
  1. Choose Elastic Cloud or self-managed, then enter your Cloud ID or node addresses.
  2. Add your API key or username and password, and a CA certificate if your self-managed cluster needs one.
  3. Add the index patterns you want investigations to query, then test the connection.
The credentials only need read access. Grant the monitor cluster privilege, and read, view_index_metadata, and monitor on the index patterns you connect: 
{
  "incident_io_role": {
    "cluster": ["monitor"],
    "indices": [
      {
        "names": ["logs-*"],
        "privileges": ["read", "view_index_metadata", "monitor"]
      }
    ]
  }
}
Each index pattern you add becomes its own data source. Enable the patterns your responders use during incidents.

​
Best practice

  • Connect the index patterns your team actually searches during incidents, rather than every index in the cluster. Each one is enabled independently, so keep the noisy or rarely-used ones off.
  • Scope names in your role to the patterns you’re connecting, so the credentials only read the indices you intend.
  • Use a dedicated read-only API key or user. Investigations only ever read from Elasticsearch.

Telemetry overview

How providers and data sources fit together.

How telemetry works

How investigations query your logs.