Prometheus

Prometheus is a time-series metric store. Nexus queries it to see how your services behaved around the time of an incident: the error rates that climbed, the latency that crept up, the saturation that tipped a service over.

​

What we support

Nexus queries Prometheus with PromQL, its query language. It goes beyond reading a metric’s raw value: PromQL turns counters and histograms into the rates and percentiles you actually reason about during an incident.

• Rates from counters. Counters only ever climb, so the raw number means little on its own. Nexus wraps them in rate() to ask the real question: how fast are requests failing right now, and was that different before the incident started.
• Percentiles from histograms. Latency lives in histogram buckets, not a single number. Nexus uses histogram_quantile() to pull out the p95 or p99 your responders care about, rather than an average that hides the tail.
• Aggregations across dimensions. With sum by, avg by, and max by, Nexus rolls a metric up to the dimension that matters: per service, per route, per namespace. That shows which slice of your fleet is misbehaving.

​

Discovering your metrics and labels

Prometheus exposes thousands of metrics and labels, and a query that names the wrong one returns nothing. Rather than guess, Nexus reads what your Prometheus actually holds, focusing on what’s been recently active: your metric names, their types, and the help text you’ve attached. It also reads your labels, with a sense of how many distinct values each one takes. Types and help text depend on your backend serving metric metadata. Many Prometheus-compatible backends, such as VictoriaMetrics, Mimir, and Thanos, don’t populate it. There Nexus works from the metric names alone. Cardinality shapes how Nexus builds queries. Grouping by a low-cardinality label like service or namespace gives a readable breakdown; grouping by a high-cardinality one like pod or instance produces noise. Nexus learns which labels are which, so it groups on the ones that clarify and filters on the ones that would overwhelm. Nexus learns this structure automatically: your metrics, their types, and your labels and their cardinality. How that works is covered in How telemetry works.

​

Connecting Prometheus

Connect Grafana, and Nexus discovers every Prometheus data source behind it automatically, using Grafana’s own credentials, with nothing separate to configure. That covers any backend Grafana fronts as a Prometheus data source: Amazon Managed Service for Prometheus, Thanos, Cortex, Mimir, and VictoriaMetrics all work the same way. Discovered Prometheus data sources start disabled, so you opt in deliberately: enable the ones your team uses from your telemetry settings. If you run several Prometheus servers (per cluster, per region, or HA replicas), you can group their data sources into one logical view: add a Prometheus aggregate from the connect wizard and pick its members. A query against the aggregate reaches every member and merges the results, so a sum or a rate is one correct global answer rather than a per-server fragment.

​

Best practice

• Connect the Grafana dashboards that query Prometheus. Nexus learns your real query patterns from them (which metrics matter, how they’re filtered and grouped), which makes Prometheus queries more accurate.
• Enable the Prometheus data sources your responders reach for during incidents, rather than every source available.

​

Related