Tempo

Tempo is Grafana’s distributed tracing backend. Investigations query it to follow a request across your services — which service handled it, where it slowed down, and where it errored.

​

What we support

Investigations query Tempo with TraceQL, its query language, in three ways:

• Span search — find spans matching a shape: slow or erroring spans by service, endpoint, attribute, or duration. This is how an investigation locates the example traces worth drilling into, without already knowing a trace ID.
• Trace retrieval by ID — pull back a complete trace once its ID is known, whether from a span search, a log line, or an error report.
• Trace-derived metrics — turn spans into time-series: request rates, counts, and latency quantiles grouped by service, environment, status, or any span attribute. You get a chart of how your traces behaved across the incident window, often for dimensions that were never recorded as a dedicated metric.

​

Reading a trace as a timeline

A raw trace is a flat list of spans. To make it useful, an investigation rebuilds the parent-child structure into a span tree and renders it as a timeline — each span shown with its service, operation, duration, and status, nested under the span that called it.

Trace: 000032595127f8cbe30fe55ccc56fb0c Duration: 144ms | Spans: 15 | Services: 3 CheckAnswerQuestion [api-gateway] (144ms) ok   └─ ValidateToken [auth-service] (12ms) ok   └─ SELECT users [database] (118ms) error

This reads the way a responder scans a trace: spot the span on the critical path, see where the time went, and see which span failed. Large traces are reduced to keep the timeline readable, but error spans and the slowest path through the trace are always kept. An investigation can then open any single span for its full attributes, and follow a link from one span to a span in another trace where a request crossed an async boundary like a queue.

​

Aggregating over span attributes

Span attributes — a customer ID, an endpoint, a SQL fingerprint, a status — are rarely exposed as labeled metrics elsewhere. TraceQL metrics let investigations chart rates and latency quantiles grouped by those attributes directly, so a question like “p95 latency on POST /v1/incidents in production” can be answered from traces even when no metric was set up to track it. Trace-derived metrics depend on Tempo’s metrics-generator; where it isn’t running, investigations fall back to span search over the same window. Investigations learn the structure of your traces — your services, the resource and span attributes you emit, and which of them are high-cardinality — automatically, so queries target attributes that actually exist. How that works is covered in How telemetry works.

​

Connecting Tempo

There are two ways to connect Tempo. Both give investigations the same access — choose whichever matches how you run it.

​

Directly

Connect Tempo on its own, with its endpoint and credentials:

• The URL of your Tempo instance.
• Any authentication it requires — for example basic auth or a bearer token, plus a tenant ID if your Tempo is multi-tenant.

​

Through Grafana

If your Tempo already sits behind Grafana, connect Grafana and Tempo is discovered automatically as one of the data sources behind it, using Grafana’s own credentials — nothing separate to configure. Either way, Tempo is disabled by default. You opt in deliberately — enable the Tempo data sources your team uses once they’re connected.

​

Best practice

• Enable the Tempo data sources your responders actually reach for during incidents, rather than every source available.
• If you want trace-derived metrics, run Tempo’s metrics-generator. Without it, investigations can still search spans and retrieve traces, but can’t chart rates or latency quantiles.
• Connect the Grafana dashboards that query Tempo. Investigations learn your real trace query patterns from them, which makes Tempo queries more accurate.

​

Related