> ## Documentation Index
> Fetch the complete documentation index at: https://docs.incident.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Tempo
> Query your Tempo traces to see how a request moved through your services during an incident.
Tempo is Grafana's distributed tracing backend. Investigations query it to follow a request across your services: which service handled it, where it slowed down, and where it errored.
You can connect Tempo directly, or have it discovered automatically when you connect
[Grafana](/investigations/connect/telemetry/grafana). Either route gives investigations the same access, so pick
whichever fits how you run Tempo.
## What we support
Investigations query Tempo with TraceQL, its query language, in three ways:
* **Span search**: find spans matching a shape, such as slow or erroring spans by service, endpoint, attribute, or duration. This is how an investigation locates the example traces worth drilling into, without already knowing a trace ID.
* **Trace retrieval by ID**: pull back a complete trace once its ID is known, whether from a span search, a log line, or an error report.
* **Trace-derived metrics**: turn spans into time-series such as request rates, counts, and latency quantiles grouped by service, environment, status, or any span attribute. You get a chart of how your traces behaved across the incident window, often for dimensions that were never recorded as a dedicated metric.
### Reading a trace as a timeline
A raw trace is a flat list of spans. To make it useful, an investigation rebuilds the parent-child structure into a span tree and renders it as a timeline, with each span shown with its service, operation, duration, and status, nested under the span that called it.
> Trace: 000032595127f8cbe30fe55ccc56fb0c
> Duration: 144ms | Spans: 15 | Services: 3
>
> CheckAnswerQuestion \[api-gateway] (144ms) ok
> └─ ValidateToken \[auth-service] (12ms) ok
> └─ SELECT users \[database] (118ms) error
This reads the way a responder scans a trace: spot the span on the critical path, see where the time went, and see which span failed. Large traces are reduced to keep the timeline readable, but error spans and the slowest path through the trace are always kept. An investigation can then open any single span for its full attributes, and follow a link from one span to a span in another trace where a request crossed an async boundary like a queue.
### Aggregating over span attributes
Span attributes (a customer ID, an endpoint, a SQL fingerprint, a status) are rarely exposed as labeled metrics elsewhere. TraceQL metrics let investigations chart rates and latency quantiles grouped by those attributes directly, so a question like "p95 latency on `POST /v1/incidents` in production" can be answered from traces even when no metric was set up to track it. Trace-derived metrics depend on Tempo's metrics-generator; where it isn't running, investigations fall back to span search over the same window.
Investigations learn the structure of your traces automatically (your services, the resource and span attributes you emit, and which of them are high-cardinality) so queries target attributes that actually exist. How that works is covered in [How telemetry works](/investigations/connect/telemetry/how-it-works#learning-your-stack).
## Connecting Tempo
There are two ways to connect Tempo. Both give investigations the same access, so choose whichever matches how you run it.
### Directly
Connect Tempo on its own, with its endpoint and credentials:
* The URL of your Tempo instance.
* Any authentication it requires, for example basic auth or a bearer token, plus a tenant ID if your Tempo is multi-tenant.
### Through Grafana
If your Tempo already sits behind Grafana, connect [Grafana](/investigations/connect/telemetry/grafana) and Tempo is discovered automatically as one of the data sources behind it, using Grafana's own credentials, with nothing separate to configure.
Either way, Tempo is disabled by default. You opt in deliberately: enable the Tempo data sources your team uses once they're connected.
## Best practice
* Enable the Tempo data sources your responders actually reach for during incidents, rather than every source available.
* If you want trace-derived metrics, run Tempo's metrics-generator. Without it, investigations can still search spans and retrieve traces, but can't chart rates or latency quantiles.
* Connect the Grafana dashboards that query Tempo. Investigations learn your real trace query patterns from them, which makes Tempo queries more accurate.
## Related
The provider Tempo can be connected through.
How investigations query your traces.