Splunk (Enterprise and Cloud Platform)

Splunk Enterprise and Splunk Cloud Platform keep your logs and metrics in indexes you query with SPL. Investigations search those indexes to read what a service was logging when an incident started, and to graph the metric that moved. Splunk Observability Cloud is a separate Splunk product with its own page, and you can connect both.

​

What we support

Connecting Splunk gives investigations two capabilities, each of which you can enable independently, plus your dashboards:

​

Logs

Investigations search your indexes with SPL to read what a service was logging at the time of an incident.

​

Trends from your logs

SPL turns events into time series, and investigations use this to graph trends straight from your logs: an error rate climbing, request volume dropping away, a message appearing right after a deploy. You get a chart of what your logs were doing even where you never sent a metric for it.

​

Metrics

Investigations query your metric indexes with SPL and graph a measurement across the incident’s time window, so a CPU saturation, a latency change, or a queue backing up shows against the period that matters.

​

Dashboards

Investigations read the searches behind your dashboard panels to learn how your team queries Splunk. The panels themselves aren’t run during an investigation. Investigations learn this structure automatically. How that works is covered in How telemetry works. Every finding cites the search behind it, and links back to that search in Splunk with the same time bounds, so you can pick up where the investigation left off.

​

Connecting Splunk

You connect Splunk directly. What you’ll need:

• Your deployment type, Splunk Enterprise or Splunk Cloud Platform.
• Your management API endpoint, for example https://acme.splunkcloud.com:8089. This is the management API, not the web interface you browse. Port 8089 is the default on both deployment types.
• A public HTTPS hostname for that endpoint. Bare IP addresses and internal hostnames aren’t accepted. If your Splunk Enterprise deployment sits inside a private network, connect it through a proxy.
• REST API access, on Splunk Cloud Platform. The management API is closed by default. Ask Splunk Support to enable it for your stack, and add our IP ranges to your stack’s REST API allow list.
• An authentication token, created in Splunk under Settings → Tokens. Token authentication has to be enabled on the deployment first, which is done from the same page. Splunk covers this in its documentation for Splunk Cloud Platform and for Splunk Enterprise.
• A role for that token with the search capability, which every Splunk query needs. The built-in user role is enough.
• The indexes in that role’s allowed list. This, not the role’s capabilities, decides what investigations can read. An index outside the list returns nothing rather than an error.
• A CA certificate (Splunk Enterprise only, optional). Provide one if your deployment uses a self-signed or internal certificate authority. We always verify TLS.

To connect:

1. Get in touch or ask in our shared Slack channel, and we’ll enable Splunk for your account.
2. From the Nexus telemetry settings, add a telemetry data source and choose Splunk.
3. Choose which capabilities (logs, metrics, or both) investigations can use.
4. Enter your deployment type, endpoint, and token, then test the connection. We check that Splunk accepts the token and that its role can run searches.
5. Choose the dashboards you want investigations to learn from.

​

Best practice

• Create a dedicated token on a role that has the search capability and an allowed index list covering the indexes you want queried. That role defines everything investigations can reach.
• Give the token a long expiry, or plan to rotate it. Splunk expires tokens, and the connection stops working the day it does.
• Connect logs and metrics together where you have both. With both available, an investigation can move from a metric that moved to the log lines explaining it.
• Select the dashboards your responders actually open during an incident. Investigations learn your team’s search conventions from them, so a dashboard that reflects how you really debug a service makes for better queries than one nobody looks at.

​

Related