Prometheus

Connect Prometheus, Mimir, Cortex, Thanos or VictoriaMetrics for PromQL reads, what the agent's prometheus_query tool returns, and how its reads are gated.

How-to · Updated 2026-10-23 · Describes release 0.1.164

Overview

A Prometheus connection gives the agents PromQL. The Incident agent reads it during diagnosis (metrics is one of the five categories that agent needs) and the FinOps agent reads utilisation baselines from it; the estate x-ray and the insights chat use the same connection. One connection also feeds two more things: the hourly service-map sync reads the Tempo service-graph metric stored in Prometheus, and a Lakehouse collection can pull metrics from it on a schedule.

The connector speaks the Prometheus HTTP API, so the same type covers Mimir and Cortex (set a path prefix and tenant ids), Thanos and VictoriaMetrics (leave both blank). Prometheus alerts do not arrive through this connection: Alertmanager posts them to the workspace webhook, see Alerts in.

Setup

Before you start

• The datasources.manage permission; the built-in owner and operator roles carry it.
• No credential in the form; a backend behind authentication takes one through the REST API (see Installation).
• The query API exposed to the platform or to a private-network agent; Installation lists the paths.

Installation

Expose the query API to the platform or to a private-network agent: api/v1/query, api/v1/query_range, api/v1/series, api/v1/labels and api/v1/status/buildinfo, plus /-/healthy for the connection check. Keep the admin API (api/v1/admin/tsdb/*) and remote write out of the exposed surface; the platform refuses them anyway, but a gateway that does not serve them is the real boundary.

The form has no credential field. A Prometheus behind authentication takes its credential through the REST API: secret_values.api_key is sent as Authorization: Bearer <token>, or secret_values.username and secret_values.password as HTTP Basic. Both are stored write-only and merged into the connection at use.

Configuration

1. Open Settings · Data in → Data sources and select Connect datasource.
2. Under Type, choose Prometheus. Enter a Connection name; the connected list shows it, and per-connection agent tools are named after it.
3. Fill the fields; the TLS-skip field sits under Advanced connection settings:

4. Optionally, open Optional · describe this environment and fill What should AIOps know about it? with what a new on-call engineer would need.
5. Under Use this connection for, choose Investigate only and select Connect and verify.

The same connection through the API (POST /api/aiops/{tenant_id}/datasources, requires datasources.manage):

{
  "type_key": "prometheus",
  "name": "prod-prometheus",
  "description": "Prod cluster metrics; checkout and payments are the critical services.",
  "config": {
    "url": "https://prometheus.example.com",
    "path_prefix": "",
    "org_ids": ""
  },
  "role": "read",
  "enabled": true
}

For Mimir, send "path_prefix": "/prometheus" and "org_ids": "team-a, team-b"; for a backend behind a bearer token add "secret_values": {"api_key": "<token>"}.

Validation

Connect and verify, and Test on the row later, sends a GET to /-/healthy and then to /api/v1/status/buildinfo. With a path prefix it sends only <prefix>/api/v1/status/buildinfo, because /-/healthy is not served under the prefix; with tenant ids it sends X-Scope-OrgID with the first tenant. You should see a dialog titled <connection name> is connected with Connected — verified from the control plane, and later in the row's Last test ok · HTTP 200 /-/healthy — Prometheus Server is Healthy.: the status, the path answered and the first 120 characters of the body. The check proves reachability and, where the backend enforces it, authentication; it does not prove that the series you expect exist. Run up through the agent, or in your own Prometheus, and confirm the intended services appear.

If something fails

The dialog shows Needs a fix, what to fix and the check's own detail; the row is saved either way. Select Edit, correct the field the detail names and save again, or select Retry once a cause outside the form (network, vendor-side permission) is fixed. Every verdict this connector produces, and what to do about it, is in Troubleshooting below.

Data collected

Tools the agent gets

• prometheus_query — bound to the first Prometheus connection in the workspace. Parameters: query (PromQL), and start, end, step for a range query; an instant query runs when neither start nor end is given. With more than one tenant id, org_id chooses the tenant (the first is the default). The response is compacted into metric{labels} = value lines: at most 50 series, the last 20 points of each range series (+N earlier points says what was cut), success: empty result set (no matching series) when nothing matched, and HTTP <code> with the backend's body on an error.
• <name>_prometheus_query — every further Prometheus connection gets its own native, named after the connection, so a fragmented estate is fully reachable.
• <name>_request — the raw HTTP tool against the base URL: method, path, query, body, content_type, and org_id when tenant ids are set. Use it for the label and series endpoints.

Responses are capped at 20,000 characters per call.

Read surface

GET and HEAD run freely, and so do POST requests to api/v1/query, api/v1/query_range, api/v1/series, api/v1/labels and api/v1/query_exemplars, matched on whole path segments so /prometheus/api/v1/query qualifies too. Everything else is refused in an investigation, never offered for approval: POST /api/v1/write, and the admin endpoints api/v1/admin/tsdb/delete_series, clean_tombstones and snapshot even over GET, because VictoriaMetrics answers a delete on GET. The refusal reads Investigation connections are read-only. This operation was not executed; prepare an incident change using a write connection and obtain approval. The full rule table is under The read/write gate in detail in Connections and remediation access.

Alerts in

Alerting rules reach the platform through Alertmanager's webhook receiver, POST /webhook/alertmanager/<workspace-slug> with the token in Authorization: Bearer and send_resolved: true, not through this connection. Every alerting rule must carry a service label: set it on the rule, or attach it in the scrape config so it rides every series. Alerts from your observability platform covers the setup; the webhook reference has the fields the adapter reads.

Service map

When Tempo's service-graph processor remote-writes traces_service_graph_request_total{client, server} into this Prometheus, the hourly trace sync reads increase(traces_service_graph_request_total[3600s]) summed by client, server, connection_type and the scope labels namespace, cluster and k8s_namespace_name, once per declared tenant id, and records caller → callee edges with origin traced. Series whose connection_type is messaging_system or virtual_node are dropped. Traced edges show on the map and widen the blast radius; they fold incidents only where the workspace opts traced edges in (aiops_topology_fold_origins).

Troubleshooting

The generic verdict strings and the SSRF guard are explained in Test a connection and read the failure. Every connection status word is in the status and error reference.

Next steps

• Select Discover this connection in the success dialog to find the services and dependencies behind it: Launch a scan.
• Send Prometheus alerts in through Alertmanager's webhook: Alerts in.
• See what else each agent needs in Choose your connection.