Platform monitoring and alerting

Scrape the control plane and runners with your monitoring system.

How-to · Updated 2026-10-27 · Describes release 0.1.164

Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).

Scrape the control plane and runners with your monitoring system. Route platform failures to an independent on-call receiver, never into the platform's own ingest: an unavailable platform cannot investigate itself.

Connect the scrape

GET /metrics is a separate listener on 9464, controlled by AIOPS_METRICS_PORT (0 disables it). The chart exposes ClusterIP Services <release>-metrics and headless <release>-runner-metrics; 9464 is excluded from the application Service, Ingress, HTTPRoute and load-balancer targets. It requires the bearer token in AIOPS_METRICS_TOKEN (the chart's secrets.AIOPS_METRICS_TOKEN, derived from AIOPS_SECRET_KEY when left empty; with existingSecret, add the key yourself) and must remain inside the cluster network. Without a token, any workspace's API connection aimed at 9464 reads every workspace's series, because the control plane dials connectors from inside the cluster.

Use the two endpoint-discovery jobs in Scrape with plain Prometheus, or enable metrics.serviceMonitor.enabled with labels and namespace selection matching your Prometheus Operator. The runner namespace is normally aiops-agents.

Set metrics.runnerScrapeFrom.namespaceSelector and podSelector to your Prometheus peer. Empty selectors leave the chart's runner port denied. Verify network-plugin enforcement: on the measured EKS Auto Mode cluster the policy objects were inert, so runners answered even before a peer was allowed. Chart configuration alone is not an isolation test. The control-plane chart has no NetworkPolicy; add a customer policy if required.

The following command is quoted from the guide's tested in-cluster check, with release and namespace normalized to aiops:

TOKEN=$(kubectl -n aiops get secret aiops-secrets \
  -o jsonpath='{.data.AIOPS_METRICS_TOKEN}' | base64 -d)
kubectl -n aiops run curl --rm -it --image=curlimages/curl -- \
  curl -s -H "Authorization: Bearer $TOKEN" http://aiops-metrics:9464/metrics | grep aiops_build_info

Check the build version and all expected targets UP. The listener must not be reachable outside the cluster. Use your existing Prometheus and external network checks to confirm the boundary; do not publish metrics through ingress.

Seven starter rules

The expressions below are the corrected guide expressions. Put them in your existing rule group or PrometheusRule and use the guide's annotations. The for durations and thresholds are starter values, not SLOs.

groups:
  - name: aiops-platform
    rules:
      - alert: AiopsRunnerSilent
        expr: max by (tenant) (aiops_runner_last_claim_age_seconds) > 600
        for: 5m
        labels: { severity: critical }
      - alert: AiopsRunnerNotReady
        expr: (max by (tenant) (aiops_runner_ready) == 0) and on (tenant) (sum by (tenant) (aiops_investigations{status="queued"}) > 0)
        for: 5m
        labels: { severity: critical }
      - alert: AiopsDbPoolExhausted
        expr: (aiops_db_pool_in_use >= aiops_db_pool_size + aiops_db_pool_max_overflow) and (aiops_db_pool_size > 0)
        for: 2m
        labels: { severity: critical }
      - alert: AiopsIngest5xx
        expr: sum(rate(aiops_http_requests_total{route=~"/webhook/.*",status_class="5xx"}[5m])) > 0
        for: 5m
        labels: { severity: critical }
      - alert: AiopsBudgetTrip
        expr: >-
          (increase(aiops_budget_trips_total{kind=~"tenant_day|tenant_month|kill_switch"}[15m]) > 0)
          or
          (aiops_budget_trips_total{kind=~"tenant_day|tenant_month|kill_switch"}
            unless aiops_budget_trips_total{kind=~"tenant_day|tenant_month|kill_switch"} offset 15m)
        labels: { severity: warning }
      - alert: AiopsLeaderLost
        expr: (sum(aiops_leader) or vector(0)) == 0
        for: 2m
        labels: { severity: critical }
      - alert: AiopsAuditSealBacklog
        expr: max(aiops_audit_unsealed_rows) > 0
        for: 10m
        labels: { severity: warning }

Keep the non-zero pool check: API-mode runners export a zero-size pool, which the bare comparison incorrectly treats as exhausted. Keep the second budget-trip clause: the first trip creates a series at 1, so increase() alone misses it. The ingest rule needs sustained errors; an isolated first 5xx does not fire it. Use the incident table for each alert's response.

With the record archive on (AIOPS_ARCHIVE_BUCKET), add one more rule to the same group: a workspace whose last good archive attempt is two days old is no longer having its expired records archived or removed.

      - alert: AiopsArchiveStale
        expr: time() - max by (tenant) (aiops_archive_last_success_timestamp_seconds) > 2 * 86400
        for: 30m
        labels: { severity: warning }

The error is on the Ops console's Data retention default card, under the workspaces whose last attempt failed: a bucket setting gone (Paused), a refused credential, or a failed batch.

Scheduler sub-tick timing

The leader's scheduler runs its sub-ticks one after another every 60 seconds, so one slow sub-tick delays all the rest, the audit seal included. aiops_scheduler_subtick_seconds{subtick} times the slow ones: the change harvest, the scanner findings pull, the estate inventory, record retention, the audit seal, the lakehouse collector, group_triage and the synthetic-check monitors. To see which one holds the tick up:

histogram_quantile(0.99, sum by (subtick, le) (rate(aiops_scheduler_subtick_seconds_bucket[15m])))

A sub-tick whose p99 approaches 60 seconds stretches every tick; if aiops_scheduler_last_tick_timestamp_seconds also falls behind, this is the place to start.

Route, dashboard and evidence

Route these rules in your external Alertmanager to the named platform on-call service, with resolved notifications. RDS free-storage/connection alarms use the customer SNS topics or the confirmed alarm_email subscription. With no recipient configured the alarms exist but page no one. Portal notification settings concern estate incidents; they are not a replacement for external platform monitoring.

Import the platform Grafana dashboard. Counters are produced by the process doing the work, so sum them across control plane and runners. Database-derived gauges are refreshed every 15 seconds by the control-plane leader; absent live-work series means zero, not necessarily a scrape failure. Check target up separately.

The 6 October verification on 0.1.105 found 14/14 targets UP, all seven rules healthy, and a first AiopsBudgetTrip delivered to a separate receiver. That receiver was an echo server, not proof that a customer's on-call integration is configured. Repeat a controlled route test in your monitoring system and retain the target/rule results plus firing and resolved delivery receipts.

Components log to stdout; forward them using the customer's log pipeline. Platform traces and additional counters for lease expiry/resume, approval decisions and database pool wait are outside the delivered monitoring scope.

Sources: Monitoring guide, verification record, and RDS settings.

Platform monitoring — scrape the AIOps platform from your Prometheus

The platform is a service your operations team runs. This guide tells you what it exports, where the metrics are reachable (only inside the cluster — by design), how to scrape them with plain Prometheus or the Prometheus Operator, and which alerts and dashboard to start from. Companion: deploy/aiops-helm/README.md for the chart values referenced here.

Which path do I need?

What is exposed — and the in-cluster-only guarantee

Every control-plane pod and every per-tenant runner pod serves GET /metrics (Prometheus text format) on a separate listener, port 9464 (AIOPS_METRICS_PORT; 0 turns it off). That port:

• is never part of the app Service (<release>), the Ingress, the HTTPRoute or any load balancer target group — those reference the app Service's http port only;
• is exposed solely by two ClusterIP Services the chart renders: <release>-metrics (control plane, release namespace) and <release>-runner-metrics (headless, runner namespace, selects every runner pod via aiops.managed=1);
• requires a bearer token whenever AIOPS_METRICS_TOKEN is set (Authorization: Bearer <token>; anything else gets 401). The chart sets it: secrets.AIOPS_METRICS_TOKEN, or, left empty, a stable value derived from secrets.AIOPS_SECRET_KEY; with existingSecret, add an AIOPS_METRICS_TOKEN key to that Secret yourself. Runner pods get the same token from the control plane. The cluster network is not enough on its own: the control plane dials workspace connectors from inside it, so without a token a workspace's API connection aimed at port 9464 reads every workspace's series. Do not expose the port through anything that leaves the cluster either;
• on runner pods sits behind the default-deny NetworkPolicy the chart ships: the port opens only for the peer you name in metrics.runnerScrapeFrom. The chart ships the policy; the cluster enforces it only when its network plugin implements NetworkPolicy — on EKS the VPC CNI network-policy agent, which is off by default (Auto Mode included). Without it the policy is inert and the port answers anything inside the cluster, so the cluster network stays the boundary either way.

Labels are deliberately low-cardinality and contain no PII: tenant slug (never an id), pillar, alert source, status, model id, HTTP route template (/api/aiops/{tid}/incidents, never the raw URL), method, status class. Requests that match no route are labelled by mount prefix (/portal, /mcp, …) or unmatched. A webhook call with bad credentials is counted under tenant="unknown" — the slug in that URL is untrusted input. Investigation series carry a bounded trigger label (alert, schedule, manual, recheck, correlate, change, compliance, xray, chat, api, else other) so interactive chat/api turns never read as incident runs — filter trigger!~"chat|api" for run-rate panels.

Which process counts what: counters increment in the process where the event happens (webhooks and HTTP on the control plane, LLM usage in the process running the agent — a runner pod in remote mode), so scrape both Services and sum() counters across replicas in queries. Gauges marked DB are computed every 15 s from the database on the control-plane leader replica only (non-leaders export none of them) and are the estate-wide truth; aiops_leader, the DB pool gauges and aiops_build_info are per replica. Benchmark runs are excluded from operational series (they are test spend).

Scrape with plain Prometheus

Only the control-plane pod carries prometheus.io/scrape|port|path annotations. Runner pods carry none on purpose: where the cluster enforces their NetworkPolicy it keeps port 9464 closed until metrics.runnerScrapeFrom names your Prometheus, so an annotation-driven pod job would only show DOWN targets — scrape runners through the headless <release>-runner-metrics Service instead. An annotation-driven job that scrapes the control-plane pod needs the same authorization block as the jobs below: annotations cannot carry a token, so without it that target answers 401 and shows DOWN.

Upgrading to a chart that sets the token: the token takes effect as the control plane restarts, so give every scrape job its authorization block before the upgrade, or expect its targets DOWN until you do. With existingSecret, the listener stays open until that Secret holds AIOPS_METRICS_TOKEN: make your Prometheus send the token first, then add the key, then set metrics.serviceMonitor.bearerToken: true if you use the ServiceMonitor. Explicit endpoints jobs need no annotations. Each job sends the metrics token: mount the AIOPS_METRICS_TOKEN key of the release Secret into your Prometheus (copy it into the monitoring namespace if Prometheus runs there) and point credentials_file at it:

scrape_configs:
  - job_name: aiops-control-plane
    authorization:
      credentials_file: /etc/prometheus/aiops/AIOPS_METRICS_TOKEN
    kubernetes_sd_configs:
      - role: endpoints
        namespaces: { names: ["<release-namespace>"] }
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: <release>-metrics
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        regex: metrics
        action: keep
  - job_name: aiops-runners
    authorization:
      credentials_file: /etc/prometheus/aiops/AIOPS_METRICS_TOKEN
    kubernetes_sd_configs:
      - role: endpoints
        namespaces: { names: ["aiops-agents"] }
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: <release>-runner-metrics
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_aiops_tenant]
        target_label: runner_tenant

Open the runner port for that Prometheus (values file):

metrics:
  runnerScrapeFrom:
    namespaceSelector:
      matchLabels: { kubernetes.io/metadata.name: <monitoring-namespace> }
    podSelector:
      matchLabels: { app.kubernetes.io/name: prometheus }

Leaving both selectors empty keeps the runner port closed (deny-all). The control-plane pod has no NetworkPolicy in the chart; add your own if your cluster policy requires one (allow TCP 9464 from the same peer).

Scrape with the Prometheus Operator

metrics:
  serviceMonitor:
    enabled: true
    labels: { release: kube-prometheus-stack }   # whatever your serviceMonitorSelector wants
    interval: 30s
  runnerScrapeFrom: { … as above … }

One ServiceMonitor (<release>, release namespace) selects both component=metrics Services across the release and runner namespaces, and sends the token from the release Secret's AIOPS_METRICS_TOKEN key. With existingSecret, set metrics.serviceMonitor.bearerToken: true once that Secret holds the key. If the operator watches only its own namespace, set serviceMonitorNamespaceSelector on your Prometheus to include the release namespace.

Metric catalogue

Counters end in _total; histograms expose _bucket/_sum/_count. DB = refreshed on the control plane from the database every 15 s.

Alert rules to start from

Drop this into a PrometheusRule (operator) or the groups: of a rule file. Placeholders: <release> is your Helm release name. Thresholds are starting points, not SLOs.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: aiops-platform
spec:
  groups:
    - name: aiops-platform
      rules:
        - alert: AiopsRunnerSilent
          expr: max by (tenant) (aiops_runner_last_claim_age_seconds) > 600
          for: 5m
          labels: { severity: critical }
          annotations:
            summary: "Runner for tenant {{ $labels.tenant }} has not heartbeated for 10m"
            description: "A claimed job stopped renewing its lease — the runner pod is dead or wedged. Its work resumes from the committed transcript once the lease expires; check the runner Deployment in the runner namespace."
        - alert: AiopsRunnerNotReady
          expr: (max by (tenant) (aiops_runner_ready) == 0) and on (tenant) (sum by (tenant) (aiops_investigations{status="queued"}) > 0)
          for: 5m
          labels: { severity: critical }
          annotations:
            summary: "Runner for tenant {{ $labels.tenant }} has no ready pod while work is queued"
            description: "The runner Deployment exists but its pod is not running — usually unschedulable (node capacity) or crashlooping. Queued investigations stay `running` at zero cost until it comes up. `kubectl -n aiops-agents describe pod` for the scheduling event."
        - alert: AiopsDbPoolExhausted
          # a runner in api mode has no pool and exports 0/0/0, which the bare
          # comparison would read as exhausted
          expr: (aiops_db_pool_in_use >= aiops_db_pool_size + aiops_db_pool_max_overflow) and (aiops_db_pool_size > 0)
          for: 2m
          labels: { severity: critical }
          annotations:
            summary: "Control-plane DB pool exhausted on {{ $labels.instance }}"
            description: "Every pooled connection is checked out; API and ingest start failing. Raise AIOPS_DB_POOL_SIZE / AIOPS_DB_MAX_OVERFLOW or lower AIOPS_MAX_CONCURRENT_INVESTIGATIONS."
        - alert: AiopsIngest5xx
          expr: sum(rate(aiops_http_requests_total{route=~"/webhook/.*",status_class="5xx"}[5m])) > 0
          for: 5m
          labels: { severity: critical }
          annotations:
            summary: "Webhook ingest is returning 5xx"
            description: "Alert deliveries are failing at the platform; senders retry, but a sustained 5xx means alerts are not becoming incidents. The 5xx series is created by the first failure, so one isolated 5xx never fires this — only a sustained run does."
        - alert: AiopsBudgetTrip
          # second half: a series that did not exist 15 minutes ago. Counters
          # are created on their first increment, so a tenant's FIRST trip
          # appears as a bare 1 and increase() alone never sees it.
          expr: >-
            (increase(aiops_budget_trips_total{kind=~"tenant_day|tenant_month|kill_switch"}[15m]) > 0)
            or
            (aiops_budget_trips_total{kind=~"tenant_day|tenant_month|kill_switch"}
              unless aiops_budget_trips_total{kind=~"tenant_day|tenant_month|kill_switch"} offset 15m)
          labels: { severity: warning }
          annotations:
            summary: "Agent spend blocked for tenant {{ $labels.tenant }} ({{ $labels.kind }})"
            description: "New investigations are failing at start and their incidents are escalated to humans. Raise the budget on the Guardrails page or lift the kill switch."
        - alert: AiopsLeaderLost
          expr: (sum(aiops_leader) or vector(0)) == 0
          for: 2m
          labels: { severity: critical }
          annotations:
            summary: "No control-plane replica is leader"
            description: "The scheduler, sweeps, provisioner and audit sealer are not running anywhere. Check the Postgres advisory-lock connection (AIOPS_LEADER_ELECTION)."
        - alert: AiopsTicketsFailing
          # every ticket call of a tenant failing for 30 minutes: an expired
          # token, a revoked user, a sleeping developer instance
          expr: >-
            (sum by (tenant, system) (increase(aiops_ticket_operations_total{outcome="failed"}[30m])) > 0)
            unless
            (sum by (tenant, system) (increase(aiops_ticket_operations_total{outcome="ok"}[30m])) > 0)
          labels: { severity: warning }
          annotations:
            summary: "Ticket filing/write-back failing for tenant {{ $labels.tenant }} ({{ $labels.system }})"
            description: "No ticket call succeeded for 30 minutes and some failed. Incidents show 'Ticket filing failed: <reason>' with Retry; the usual cause is an expired API token or PAT — Test the connection on the Connections page."
        - alert: AiopsAuditSealBacklog
          expr: max(aiops_audit_unsealed_rows) > 0
          for: 10m
          labels: { severity: warning }
          annotations:
            summary: "Audit chain sealer has a backlog"
            description: "Audit rows have stayed unsealed for 10 minutes; the tamper-evident chain is not advancing. Usually the leader's scheduler tick is stalled — see aiops_scheduler_last_tick_timestamp_seconds."
        - alert: AiopsArchiveStale
          # a workspace on the record archive with no good attempt for two days:
          # retention is not removing (nor archiving) its expired records
          expr: time() - max by (tenant) (aiops_archive_last_success_timestamp_seconds) > 2 * 86400
          for: 30m
          labels: { severity: warning }
          annotations:
            summary: "Record archive stalled for tenant {{ $labels.tenant }}"
            description: "No archive attempt has completed for two days. The Ops console's Data retention default card names the error under the workspaces whose last attempt failed (bucket gone = Paused, a refused credential, a failed batch)."

Dashboard

deploy/grafana/aiops-platform-dashboard.json (Grafana schema v39, one DS_PROMETHEUS datasource input) — import via Dashboards → New → Import. Panels: webhook rate by outcome, incidents opened, live investigations by status, finished outcomes, LLM spend by model, runner liveness, pending approvals, HTTP latency p95 by route, DB pool, audit seal backlog.

Verify from inside the cluster

Verified on the hosted sandbox on 2026-10-06 (build 0.1.105): every target UP, the seven rules above evaluating, and one AiopsBudgetTrip delivered to a non-agent receiver — platform-monitoring-verification-2026-10.md.

TOKEN=$(kubectl -n <release-namespace> get secret <release>-secrets \
  -o jsonpath='{.data.AIOPS_METRICS_TOKEN}' | base64 -d)
kubectl -n <release-namespace> run curl --rm -it --image=curlimages/curl -- \
  curl -s -H "Authorization: Bearer $TOKEN" http://<release>-metrics:9464/metrics | grep aiops_build_info

Without the header the same call answers 401, and from outside the cluster the path must fail altogether — that is the contract.

Note: in remote-runner (api) mode the control plane folds an operator cancel and a tenant pause into one stop signal, so a per-run cancel there is booked as kind="kill_switch"; embedded and db modes book kind="cancelled".

Next steps

• Next: Security operations
• All Install and operate the platform pages
• Documentation home