How telemetry works

Querying an unfamiliar observability stack well is hard, even for experienced engineers — you have to know which system holds the answer, the query language it speaks, and the labels and conventions specific to your setup. Telemetry is the part of investigations that does all of this for you. This page explains how it works: how an investigation picks the right data source, writes a correct query against it, and gets better at your stack over time.

​

Choosing where to look

A connected data source advertises what it can do through its capabilities — logs, metrics, traces, SQL, dashboards. A question about error rates routes to the data sources that can answer it with metrics; a question about what a service logged routes to the ones that hold logs. The investigation never has to query a system that couldn’t possibly answer. To choose between data sources that share a capability, the investigation uses a short, learned summary of what each one is actually good for in your environment — so “payment errors” goes to the data source that holds payment logs, not just any log source. For providers like Grafana, the same routing picks the right child behind it: the relevant Loki source for logs, Prometheus for metrics, and so on. When several lines of enquiry are independent, the investigation runs them in parallel — pulling logs and metrics at the same time rather than one after another.

​

Writing the query

Once it knows where to look, the investigation translates your question into a real query in that data source’s own language — LogQL for Loki, PromQL for Prometheus, SQL for a database, the right trace query for your tracing backend. This is the job of a planner. A planner doesn’t write queries from generic knowledge of a query language. For the data source it’s targeting, it works from:

• A query language reference — how to write correct, efficient queries in that language, including how to recover from timeouts and expensive queries.
• That data source’s guidance — the real labels, fields, and conventions discovered in your instance, so it filters on attributes that actually exist (see below).
• Proven patterns — example queries that have worked before for similar questions, used as a starting point rather than guessed from scratch (see memory).

Before a query runs, it’s checked — including parsing it with the data source’s own query parser to catch invalid syntax. If a query does fail when it runs — a syntax error, a timeout, a cardinality limit — the investigation reads the error and rewrites the query to address it, rather than giving up or repeating the same mistake.

​

Learning your stack

The reason an investigation can query your systems well is that it learns the shape of each one. For every connected data source, we continually build guidance — a living description of how to query that specific data source in your environment. It comes from three sources:

• Probing — we introspect the data source directly for the ground truth: which labels and fields exist, their cardinality, the kinds of events or metrics it holds.
• Your dashboards — the dashboards your team has built are effectively a record of the questions you ask most and the queries that answer them, so we learn your real query patterns from them.
• Proven queries — patterns from past queries that successfully answered a question feed back in (see below).

Guidance is generated automatically when a data source is connected and kept fresh as your systems change. It’s what lets a planner filter on the labels you actually use and reach for the right defaults when a question is vague — the difference between a query written by someone seeing your stack for the first time and one written by someone who already knows it.

​

Getting better over time

Beyond guidance, investigations keep a memory of what works: a learned library of proven question-to-query patterns for each data source, built from the queries that successfully answered questions during real investigations. When a planner writes a new query, it draws on the memories most similar to the question as worked examples, so it reaches a correct query faster and more reliably — and those patterns also enrich the guidance above. Memory grows from your own investigations, so the agent gets better at your stack the more it’s used — see Memory for how it’s built and how it compounds.

​

Making sense of the results

Conventional observability tools weren’t built to hand their data to a language model. Pipe megabytes of JSON log lines straight into a model — as a direct MCP connection does — and the result is slow, expensive, and unreliable: models don’t reason well over that much repetitive text, so the few lines that matter get lost. Raw time-series data is just as hard for a model to read — a wall of timestamped numbers says little about the shape of what actually happened. So we don’t pass raw output to the model. We’ve built our own formats for each kind of telemetry — logs, metrics, and traces — designed around how models read. They cut the repetition (shared fields stated once, runs of similar lines collapsed to just what changed between them), describe the shape of the data rather than every point in it, and render traces as readable timelines. What reaches the model is enough for it to spot the pattern, the anomaly, or the change — without burying it in everything else.

​

Related