Pyroscope

Pyroscope is Grafana’s continuous profiling backend. Investigations query it to see where your services were spending CPU and allocating memory around the time of an incident — broken down by the functions and call paths responsible.

​

What we support

Profiling answers a question logs, metrics, and traces can’t. A metric tells you CPU hit 90%, a trace tells you which request was slow, but neither tells you which function inside the process was burning the cycles. Pyroscope does: it samples your running services continuously, so investigations can look back at the incident window and see exactly which code paths were responsible. Investigations query Pyroscope by profile type and a label selector — for example, the CPU profile for {service_name="api"} over the incident window. Profile types depend on what your services emit, and commonly include:

• CPU — where processes spent their time on the CPU.
• Memory — allocations and in-use memory, by where they were allocated.
• Goroutines and other types — concurrency and other resource profiles, where your services produce them.

​

Reading a flame graph as findings

A raw profile is a flame graph: thousands of stack frames that are hard to read without clicking around. Investigations turn one into two things you can act on:

• Top functions — the functions that consumed the most CPU or memory in their own right, ranked, with each one’s share of the total.
• The hottest path — the single heaviest call chain from entry point down to the leaf, so you can see not just which function was expensive but how the code reached it.

This is the part that’s normally tedious to do by hand during an incident, done for you: a profile narrowed to the few frames that explain where the cost went. Investigations learn the profile types, labels, and service names in your Pyroscope instance automatically, so they query for profile types you actually emit and filter on labels that exist. How that works is covered in How telemetry works.

​

Connecting Pyroscope

Connect Grafana, and Pyroscope is discovered automatically as one of the data sources behind it, using Grafana’s credentials — nothing separate to configure. Once connected, Pyroscope is disabled by default. Enable the Pyroscope data sources your team uses once they’re available.

​

Best practice

• Enable Pyroscope for the services whose CPU and memory behavior you investigate, rather than every source available.
• Connect the Grafana dashboards your team uses for profiling. Investigations learn your real query patterns from them, which makes Pyroscope queries more accurate.

​

Related