# Platform configuration > Shared configuration for customer workspaces, including connector types, default models, user accounts, admin MCP, the audit log and release verification. _Reference · Updated 2026-10-26 · Describes release 0.1.164_ ## What this chapter is for The platform console (`/ops`) holds the configuration that is **shared by every tenant**: which connector types exist at all, which model each capability tier maps to before a tenant sets its own, who has an account on this deployment, and the append-only record of every mutation across all tenants. Customer organizations, workspace assignments, support access, runner tokens, Keycloak provisioning and workspace suspension are covered in [Customers and workspaces](/help/ops-tenants.md). All product features are included for valid enabled workspaces; organization assignment is separate from feature availability. Model gateway setup is in the [Connecting the LLM gateway](/help/ops-llm-gateway.md) chapter. Workspace-owned datasources, agents, guardrails and roles are documented in the separate portal handbook at `/portal/docs`. ## Who can use this console, and what it does not grant The console itself is gated on the **platform superadmin** flag. An authenticated account without it sees "This console is for platform staff", the line "Your account is not a superadmin. Use the tenant portal instead." and a **Go to the portal** button, on every page. Behind the console, the API enforces superadmin independently on every write in this chapter, and on the audit, users, runner and tier-default reads — a hand-crafted request from a tenant account is refused with the same rule the screen applies. One exception is deliberate: `GET /api/aiops/datasource-types` and `GET /api/aiops/datasource-categories` are readable by **any authenticated user**, because tenants need the catalog to connect their own datasources. They cannot write to it. Nothing in this console is reachable through a tenant RBAC permission key, and the superadmin flag grants **no tenant data**. A superadmin has no incidents, no investigations, no alerts and no datasource configuration. Staff who need to see inside a tenant take an expiring, audited support-access grant instead, which is documented in the **Customers and workspaces** chapter. Platform administration requires **superadmin**. Workspace-scoped actions, such as rotating a webhook token, also retain their normal workspace permissions. Neither an organization role nor a platform role bypasses that boundary. The console separates **Overview** (`/ops/`), **Customers** (`/ops/customers`) and **Workspaces** (`/ops/workspaces`). Customers are actual organization records; workspaces are the Tenant records that isolate operational data. **Users**, **Billing**, configuration pages and **Documentation** remain separate destinations. Old `/ops/groups` and `/ops/tenants/` links remain supported aliases. ## Concepts **Connector type vs. datasource instance.** A *type* is a catalog row: this is what a Prometheus looks like, here are the fields a tenant must fill, and here is how those values render into a tool. An *instance* is one tenant's connected copy of that type, with its own URL, credentials and name. Types are platform-owned and live in this chapter; instances are tenant-owned and live in the tenant handbook's *Connecting your estate* chapter. **Builtin vs. custom types.** The 36 types shipped in the product are **code-owned**. On every start the platform re-syncs their `name`, `description`, `kind`, `fields_schema` and `config_template` from code, so a newly shipped field reaches existing databases without a migration. `enabled`, `default_readonly` and `pillars` are never re-synced — those are your toggles, and seeding leaves them alone forever. The API therefore refuses structural edits to a builtin, because a 200 followed by a silent revert on the next deploy is a trap. Custom types — keys you create — are never touched by seeding at all. **Tier map inheritance.** Work is routed by *tier*, not by model id: estate x-ray scans use `premium`, the root cause analysis (RCA) wrap-up uses `reasoning`, the investigation loop and insights chat use `standard`, and bulk work uses `light`. The platform default map you set here is what every tenant routes on until it configures its own — **inside the control-plane process**. In a `remote` deployment that is not the whole story; the *Platform defaults* section below states the caveat in full, and it is the single most misread behavior on this page. **The write floor is not configurable from here.** Nothing in this chapter can make an agent's writes unattended. The tool layer classifies each call as a read or a write from its own table, independently of anything an operator types. On the shipped default the agent's write is refused mid-diagnosis and returned to you as a remediation plan; your platform operator can switch the fleet to pause mid-run instead. Either way no write reaches your estate without a named human. You are that platform operator: the switch is `AIOPS_DIAGNOSIS_WRITES`, `deny` (shipped default) or `pause` (the legacy mid-run stop). Estate x-ray is stricter than both and refuses a gated call outright, in every mode. A tenant's pillar configuration can only *add* gates on top of the floor. **Default readonly** in the catalog is a label — it is **not** the gate, and it does not feed the classifier. ## Connector catalog Route: `/ops/catalog`. Nav: **Connector catalog** under **Config**. API: `/api/aiops/datasource-types`. Requires platform superadmin for every write on this page. ![The connector catalog: every datasource type tenants can connect, with its kind, state, server-derived categories and pillar allowlist.](/help/media/ops/ops-catalog.png?v=0.1.132) ### What the page shows | Column | Meaning | |---|---| | **Connector** | Display name over the immutable `key` (mono). The key is the primary key of the row and what a tenant instance points at. | | **Kind** | `api`, `cli` or `mcp` — how the connector talks, and therefore which tool it becomes. The 36 shipped types are 31 `api`, 3 `cli` and 2 `mcp`. | | **State** | `enabled` or `disabled`, plus a `readonly` badge when **Default readonly** is set. | | **Category** | Server-derived grouping used by the tenant connect screen and the pillar-coverage matrix. **A type can be in several** — `aws` renders as `metrics`, `logs`, `traces`, `cloud`, `security-findings` and `cost`, one badge each. Not editable: it comes from a fixed map keyed on the connector key. | | **Pillars** | Which pillar agents may use this type. Blank renders as `all`. | | **Description** | One-line operator context, shown to tenants when connecting. | The fifteen categories are `alerting`, `metrics`, `logs`, `traces`, `dashboards`, `kubernetes`, `cloud`, `git/iac`, `itsm`, `database`, `messaging`, `on-prem`, `security-findings`, `cost` and `other`. A key that is not in the map resolves to `other`. The search box filters the loaded rows and any column header sorts them. The catalog is fetched whole and never paginated, so what you search is always the complete set. ### Add a connector type Requires platform superadmin. 1. Open `/ops/catalog` and click **New type**. 2. Enter **Key** — the identity of the row forever. It cannot be changed later. Use lowercase letters, digits, `_` and `-`, starting with a letter or digit, at most 64 characters: the server refuses any other key. 3. Enter **Name** — what tenants see in the connect picker. 4. Choose **Kind**: `api` becomes a `_request` HTTP tool, `cli` becomes a `_exec` command tool, `mcp` becomes proxied MCP tools. 5. Fill **Description** — the tenant-facing one-liner in the connect picker. Write it for a human choosing a connector; it does not become the agent's tool description (see the note below). 6. Fill **Pillars (comma-separated, blank = all)**, for example `incident, secops`. 7. Fill **fields_schema (JSON)** — a JSON array of the fields a tenant must supply. 8. Fill **config_template (JSON)** — a JSON object with `${field}` placeholders filled from the instance's config and secrets at tool-build time. 9. Leave **Enabled** ticked to make it connectable immediately, or untick it to stage the type without exposing it. 10. Leave **Default readonly** ticked unless the type exists to carry a write credential. 11. Click **Create type**. It worked when the dialog closes and a new row appears in the table with your key in mono under the display name, `enabled` in **State**, and the categories the server derived. `key` and `name` must be non-empty: the dialog checks both before sending, and the server refuses a blank name, or a key outside the rule in step 2, with a 422 validation error that the dialog prints as JSON (for example `string_pattern_mismatch` on `key`). A duplicate key is refused inline with `datasource type '' already exists`. > **Where the agent's tool description comes from.** The per-tool "Operator notes: …" line, truncated to 400 characters, is rendered from the **instance's** description — the note a tenant writes on its own connected datasource — not from the type description you set here. Your type description reaches an agent only through the estate x-ray `platform_context` tool, whose `catalog` section lists every type's name, kind, description and fields. If you want an agent to read something at the tool-picking moment, put it in the `hint` key of a `cli` `config_template`, or ask tenants to fill the instance description. ### What the operator fills in: fields_schema `fields_schema` is a JSON array. Each entry is an object, and only `name` is mandatory. | Key | Effect | |---|---| | `name` | The placeholder this field fills — `${name}` in the template. | | `label` | What the tenant sees on the connect form. Falls back to the name. | | `placeholder` | Greyed example text inside the input. Use it for the shape of the value, never for a real value. | | `secret` | `true` stores the value **write-only** in the tenant secret store and never renders it back. The tenant form labels it "(secret, write-only)", and adds "— blank keeps current" when editing an existing datasource, so a blank field on edit keeps the stored value and a filled one rotates it. | | `format` | `yaml`, `json` or `pem`. Two effects: the tenant form renders a multi-line **Textarea** instead of a single-line input, and the value is validated server-side at save — YAML must parse to a mapping, JSON must parse, a PEM value must contain a `-----BEGIN` block. A kubeconfig flattened by a single-line paste is rejected on the form, not mid-investigation. | ```json [ {"name": "url", "label": "Base URL", "placeholder": "https://metrics.example.com"}, {"name": "api_key", "label": "API key", "secret": true}, {"name": "kubeconfig", "label": "Kubeconfig", "secret": true, "format": "yaml"} ] ``` Secret values are held in a per-tenant secret store behind a `secret_ref` indirection, encrypted with Fernet when `AIOPS_TENANT_SECRET_KEY` is configured and stored in plaintext with a loud startup warning when it is not. Set that key before any real credential is pointed at the deployment. The tenant-facing half of secret handling — rotation, what is echoed back, what leaves the network — is in the tenant handbook's *Connecting your estate* chapter. The store key itself can be rotated. Use the supplied [Customer deployment runbook, §12.1 The tenant secret-store key](/help/ops-installation-runbook.md#12-1-the-tenant-secret-store-key), for the census, two approved deploys, checks, stop rules, rollback boundaries and old-key retention for backups. Do not rotate by simply overwriting the current key. The platform audit's `secrets.rekey_completed` summary and each workspace's `secret.rekeyed` records are the evidence of re-keying; confirm the runbook's clean result and connection tests before retiring a key. This console records the evidence; it does not perform the rotation. ### How those values build the request: config_template The template is rendered per instance with `${field}` substitution using `string.Template.safe_substitute`. Unknown placeholders stay **literal** rather than raising, and the tool builders then skip any value still holding an unrendered `${…}` — so a blank optional field degrades instead of breaking the whole registry build. ```json {"enabled": true, "config": {"url": "${url}", "api_key": "${api_key}", "auth_header": "${auth_header}"}} ``` For `api` types, the rendered `config` block is read for: - `url` or `base_url` — required. Without one, no HTTP tool is registered at all and a warning is logged. - `headers` — an explicit dict. Wins over every other auth key. - `api_key` or `token`, combined with `auth_header` (default `Authorization`) and `auth_scheme`. With no scheme set, `Authorization` gets `Bearer` and a custom header gets the raw token. An explicit empty scheme always sends the raw token. - `username` plus `password` — HTTP Basic. - `insecure` — skip TLS verification when the value is `1`, `true`, `yes` or `on`. Logged as a warning on every registry build. Seeding injects this field and its template entry into every shipped `api` type that has a `url` field, so a self-signed on-premise endpoint is connectable without a custom type. - `openapi_path` — points the agent at a schema it should read before calling endpoints. - `org_ids` — multi-tenant backends addressed by `X-Scope-OrgID`. A list, or comma/pipe/space separated. - `path_prefix` — URL path prefix for Prometheus-API variants. `/prometheus` for Mimir and Cortex; blank for vanilla Prometheus, Thanos and VictoriaMetrics. - `gate_paths` — operator **tighten**: force these paths to be treated as writes even on GET. Always honored. - `read_post_paths` — operator **widen**: treat these POST paths as reads. Validated before use — any token naming a write, exec or mutation surface (`write`, `exec`, `delete`, `admin`, `bulk`, `drop`, `run`, `create`, `update`, `restart` and others) is rejected, and the widen never touches PUT, PATCH or DELETE. ```json {"config": {"command": "kubectl", "args": ["--kubeconfig", "{file:kubeconfig}"], "readonly_verbs": ["get", "describe", "logs", "top", "explain"], "files": {"kubeconfig": "${kubeconfig}"}, "env": {"KUBECONFIG": "{file:kubeconfig}"}, "hint": "cluster-scoped read-only credential"}} ``` For `cli` types, `readonly_verbs` is the free list and every other verb is gated; `files` materializes named contents as 0600 files referenced by `{file:}` tokens in `args` and `env`; `env` adds process environment; and `hint` is appended to the tool description the agent reads. Commands are executed without a shell, so no template value can inject one. For `mcp` types the template is the client config, for example `{"mode": "streamable-http", "url": "${url}"}`. Both textareas are validated in the browser before anything is sent: `fields_schema` must parse as a JSON **array**, `config_template` as a JSON **object**. Parse errors appear inline in the dialog and no request is made. ### Edit a connector type Requires platform superadmin. This procedure works on **custom** types; for a builtin, read the next section first. 1. Click **Edit** on the row. 2. Change the fields you need. **Key** is not editable and is not shown. 3. Click **Save type**. It worked when the dialog closes and the row redraws with your changes. Template changes take effect on the **next agent run**, because rendering happens at registry build time: nothing rewrites a tenant's stored configuration, and no tenant has to re-enter credentials. ### Builtin types are structurally read-only The API rejects an edit to a builtin's `name`, `description`, `kind`, `fields_schema` or `config_template` with a 409 naming the blocked fields, because startup would revert them on the next deploy. **Enabled**, **Default readonly** and **Pillars** are permitted by the API. The console's editor sends **all eight** fields on every save, including the five structural ones. The API refuses only a structural field whose value changed, so on a builtin type a save that changes **Enabled**, **Default readonly** or **Pillars** goes through, and a save that also changed a structural field returns the 409 naming it and changes nothing. Over the API, send only the field you want: ```bash curl -X PATCH https://aiops.example.com/api/aiops/datasource-types/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"enabled": false}' ``` The same call takes `{"default_readonly": false}` or `{"pillars": ["incident", "secops"]}`. Mint the personal access token from your own account's token page; the account must carry the superadmin flag. It worked when the response body is the full type row with your field changed, and the table shows the new **State** after a refresh. To change a builtin's *structure*, clone it to a custom key and disable the original. ### Disable a connector type Disabling is the safe alternative to deletion. Requires platform superadmin. 1. Click **Edit**, untick **Enabled**, click **Save type**. This works for builtin and custom types alike; over the API, send the `PATCH … {"enabled": false}` call above. 2. Refresh `/ops/catalog`. It worked when the row's **State** reads `disabled`. From that moment the type disappears from the tenant **Connect datasource** picker, and every existing instance of it is skipped when an agent registry is built, so its tools vanish from the next run. The instance rows, their operator notes and their stored secrets survive untouched — re-enabling restores them exactly. ### Delete a connector type Requires platform superadmin. 1. Click **Delete** on the row. 2. Read the consequence line in the **Delete type — ``** dialog. 3. Click **Delete type**. It worked when the dialog closes and the row is gone from the table. The in-use check runs on the server *after* you confirm. If any tenant anywhere has an instance of this type, the delete is refused with `tenant datasources of this type exist; disable it instead`, shown inline in the dialog. That is deliberate: deleting the type would orphan tenant instances and their stored secrets. The check is cross-tenant, and you cannot see from this page which tenants hold them. Deleting a **builtin** key is allowed and takes effect immediately — but the next control-plane start re-creates it from code, with the shipped defaults for `enabled`, `default_readonly` and `pillars`, which means your disable is silently undone too. Disable builtins; delete only custom keys. ### "Default readonly" is a label, not a gate The flag does less than the name suggests. Today it is exactly two things: - the source of the `readonly` badge in this console's **State** column, and - a field carried in the tenant configuration an API-mode runner fetches from the control plane. It is **not** read by the tool layer. It does not appear in any tool description, and it does not influence the read/write decision for a single call. Those come from the intrinsic classifier — per connector type, and per request path, method and body — and, for `cli` types, from `readonly_verbs` in the `config_template`. Treat **Default readonly** as documentation of intent for whoever reads the catalog. The enforcement is elsewhere and cannot be switched off from this console. ## Platform defaults Route: `/ops/platform`. Nav: **Platform defaults** under **Config**. API: `/api/aiops/admin/model-tiers-default`. Requires platform superadmin. The first card is **Default model tiers**: the map every tenant inherits when it has not configured its own. Below it sit **Customer connection setup** (next section), the **Model price map** (where every price comes from, **Fetch list prices**, mappings to each publisher's model, the publisher catalogue, workspace endpoints and the editor — see [Cost, tokens and the pricing map](/help/ops-llm-gateway.md#cost-tokens-and-the-pricing-map)), the email relay, **Automatic analysis defaults** and **Data retention default** (both below). ![The platform default tier map: the four capability tiers every tenant inherits until it configures its own.](/help/media/ops/ops-platform-defaults.png?v=0.1.132) ### Set the default tier map Requires platform superadmin. 1. Open `/ops/platform`. 2. For **premium — x-ray scans**, pick a model id from the dropdown, or choose **Custom model id…** and type one. 3. Repeat for **reasoning — RCA wrap-up**. 4. Repeat for **standard — investigation loop + chat**. 5. Repeat for **light — cheap bulk work**. 6. Click **Save platform defaults**. It worked when the green line reads "Platform default tiers saved — applies to every tenant." and the environment-variable notice below the fields is gone. The save writes one `PlatformSetting` row and updates this process's routing cache, so it applies immediately to routing decisions **made in the control-plane process**. No restart is needed there. ### The model dropdown The list is fetched from your configured gateway's `/models` endpoint and each id is prefixed `openai/` so the value is directly usable by the routing layer. If no gateway is configured, or it is unreachable, the page shows "Gateway model list unavailable (…)" and you use **Custom model id…** instead. A missing gateway list never blocks saving, and a model id you type by hand is treated exactly like one you picked. Configuring the gateway itself — the endpoint, the key, what runs when neither is set — is in the [Connecting the LLM gateway](/help/ops-llm-gateway.md) chapter. If this dropdown is empty on a deployment you believe is configured, that chapter is where the diagnosis lives, not this one. ### An empty tier resolves to the nearest configured one A tier left blank is **not** an error and **not** "no model". Its picker reads "Not set — nearest configured tier". At routing time an empty tier resolves to the nearest configured tier **above** it (more capable) and, failing that, to the nearest **below**. Setting only `standard` therefore gives you a working four-tier platform where every tier resolves to that one model. Blanking a field and saving **clears** that tier. The endpoint is a merge-patch in which an empty string removes the entry, and the console sends all four keys on every save, so what you see is what the map becomes. A cleared map also empties the escalation ladder. Mid-run tier escalation and budget downgrade only have steps to take when at least two distinct models are configured; with one model, or none, the ladder is empty and no switching happens. ### The full fallback chain This is the authoritative order for the whole product. For one unit of work, in order: 1. An explicit per-agent or per-scan model override, set by the tenant, wins outright — and empties the ladder. 2. The **tenant's own** tier map, if it has any non-empty entry, resolved by the nearest-tier rule. The tenant map is used whole and is never merged with the platform default: a tenant that sets one tier owns the entire map, which keeps its spend predictable. 3. The tenant's legacy single-model setting, if it has one and no tier map. 4. This page's **platform default** map, resolved by the nearest-tier rule — on runners too, see below. 5. The `AIOPS_MODEL` environment variable. 6. The deterministic mock model. The platform stays fully clickable with no model configured at all. The tenant handbook's *Tuning the agents* chapter states step 1 and step 2 for tenants and points here for the rest, so keep this list as the only full copy — two copies of a fallback chain diverge on the next routing change. ### How your default reaches a tenant's agent runs The control plane keeps the map in a per-process cache, loaded from the database at startup and replaced the moment you save here. Jobs that run in that process — `embedded` mode, the application default when nothing sets `AIOPS_EXEC_MODE` — use it from the next call. Per-tenant runners (`AIOPS_EXEC_MODE=remote`, the Helm chart default) follow the same map from 0.1.117: - An **api-mode** runner the platform provisioned (the chart's default `dbMode`) has no database to read. The control plane sends this page's effective map with every job's configuration, so a save here applies to the next job it starts — no restart, nothing to set on the runner. A **private-network agent the customer runs** does not receive this map: it keeps its own `llm.model`, because this page names the platform gateway's model ids. - A **db-mode** runner re-reads the saved map from the database each time it polls for work. With no map saved here it falls back to its own `AIOPS_MODEL_TIERS_DEFAULT`; the provisioner copies the control plane's value onto every db-mode runner it creates. A tenant's own tier map still wins over this page on every runner, as in the chain above. A tenant whose private-network agent calls the customer's own LLM endpoint should set one that names that endpoint's model ids: investigations on that agent use it (else the agent's `llm.model`), and x-ray scans and chat resolve through this chain on the control plane before they reach the agent. A runner's own `AIOPS_MODEL_TIERS_DEFAULT` is only its fallback: an api-mode runner uses it when the control plane sends no map — the platform has none (neither this page nor the control plane's environment sets one), or the control plane is older than 0.1.117. Runner images older than 0.1.117 do not follow this page: an api-mode runner never receives the map and runs a tenant with no tier map of its own on its own `AIOPS_MODEL_TIERS_DEFAULT`, else on `AIOPS_MODEL`, and a db-mode runner reads the map only when it starts. The tenant sees no error — the runs simply use another model than this page shows, and tier escalation never fires. For a private-network agent you cannot upgrade yet, set `AIOPS_MODEL_TIERS_DEFAULT` in its environment to the same map, or give the tenant its own tier map on its **AI models** page in the portal; a tenant map travels with every job. ### The environment-variable fallback When no default has ever been saved here, the page says so and names the environment fallback. The notice names the variable you set, `AIOPS_MODEL_TIERS_DEFAULT`, and shows the values in effect, so you can read the live map without a shell. Its format is a comma-separated list of `tier=model` pairs: ```bash AIOPS_MODEL_TIERS_DEFAULT="premium=openai/big,reasoning=openai/big,standard=openai/std,light=openai/small" ``` Saving on this page takes precedence over the environment variable from that moment on: the control plane loads the saved map at every start, and runners follow it as described above. Clearing all four tiers and saving hands routing back to the variable, on the control plane and on the runners that follow this page, also after a restart; the page then shows the environment notice again, with the map now in effect. Unknown tier names and empty values in the variable are ignored rather than crashing routing, so a typo costs you one tier, not the platform. When neither this page nor `AIOPS_MODEL_TIERS_DEFAULT` sets a tier map, every tier runs on `AIOPS_MODEL`, and the notice names that model. With `AIOPS_MODEL` unset as well, embedded and db-mode runners fall back to the scripted mock responder, and api-mode runners refuse to run. ### Automatic analysis defaults API: `/api/aiops/admin/auto-investigate-defaults` (`GET`, and `PATCH` with `p1p2` and/or `catchup` as JSON booleans). Requires platform superadmin. The **Automatic analysis defaults** card holds the two switches every workspace follows for pulled Security Hub and Defender findings until it sets its own: | Switch | What it does | Shipped default | |---|---|---| | **P1/P2 findings** | Analyse a newly pulled Security Hub or Defender finding at P1 or P2 without an Analyse click. | On | | **Backlog catch-up** | Also analyse already-held P1/P2 findings, oldest first, 3 per workspace per minute, one run each. Only findings still open are covered, and only while P1/P2 analysis is on for that workspace. | Off | A tick or a cleared box saves at once; the notice reads, for example, "Backlog catch-up default on: applies to every workspace without its own switch." The control plane applies it to the next pulled finding and the next catch-up sweep, without a restart. Each save writes one `PlatformSetting` row (`auto_investigate_defaults`) and one platform audit row (`platform.auto_investigate_defaults`, before and after). **A workspace override wins over the default.** A workspace owner sets one on **Guardrails & cost** in the portal; you can set one on the workspace page (**Automatic analysis** card, chapter **Customers and workspaces**). A workspace with an override keeps it when you change the default here; one that follows the default changes with it. Every analysis is a paid agent run, and switching the catch-up on can start one run per held finding in every workspace that follows the default, so check workspace cost limits before you do. The default is held in the control plane's memory: it is loaded at start, re-read on every scheduler tick and replaced at once by a save here. Only the scheduler leader re-reads it on each tick, so with more than one control-plane replica another replica can keep an older default until it restarts. ### Data retention default API: `/api/aiops/admin/retention-defaults` (`GET`, and `PATCH` with `days` as a JSON integer from 30 to 3650 and an optional `action`, `archive` or `delete`). Requires platform superadmin. The **Data retention default** card on **Platform defaults** holds the number of days a **closed** record is kept in every workspace that has not set its own window: a resolved incident with its alerts, runs, validations, approvals, feedback, merge proposals and artifacts; a closed problem; a completed, rolled-back, failed, rejected or recorded change with its evidence and the verify and planning runs it names; a cleared finding with its risk decisions; a finished compliance or estate x-ray scan with its results, phase runs and the attachments no kept scan reuses; a chat session with its turns; and a finished run that belongs to no incident and no kept record names — any trigger except chat (the session), x-ray and compliance (the scan), benchmark (the Test Lab) and recheck (the validation row or the change). The sweep works in whole UTC months: its cutoff is the first instant of the month in which "now minus the window" falls. Shipped default: 365 days (`AIOPS_RETENTION_DAYS_DEFAULT`); the saved value wins over the environment variable. Each save writes one `PlatformSetting` row (`retention_defaults`) and one platform audit row (`platform.retention_defaults`, before and after). Nothing open is ever removed — unresolved incidents, changes that have not ended, active findings, running scans — and neither is a record with a run still running or paused or with a run or validation at or after the cutoff (a later sweep takes it whole), a cleared finding with a risk decision in force or awaiting sign-off (nor the change that decision names), the audit chain, knowledge items, skill proposals, attestations and the evidence they cite, posture snapshots, benchmark and Test Lab rows, certification artifacts or any configuration. A kept row's pointer to a removed one is nulled (skill proposal source run, intake receipt and finding incident). Before anything goes, in the same transaction, every month a removed run, validation or scan was billed in is sealed as a usage statement (table `usagestatement`, one row per workspace and month, written once); the invoice (`/admin/billing`, where a sealed tenant row carries `sealed_at`), the workspace statement, the organization view and every CSV read that row for the month from then on, so the invoice stays the same after the rows are gone. Only whole months that have ended are swept, so a month is closed before it is sealed. **A workspace override wins over the default.** A workspace owner sets one on **Guardrails & cost** in the portal (`retention_days`, 30 to 3650); a workspace with an override keeps it when you change the default here, one that follows the default changes with it on its next sweep. The sweep runs once a day per workspace inside the scheduler tick, leader-only, and can be switched off platform-wide with `AIOPS_RETENTION_SWEEP_ENABLED=false` (the portal's preview and manual sweep keep working). A sweep writes one `retention.swept` audit row for the workspace whenever it removes anything, with the counts per kind and the months it sealed; the **Data retention** card on **Guardrails & cost** shows the last one. A workspace whose sweep fails is logged once (`record retention sweep failed for workspace `) and stamped in the `retention_last_sweep` platform setting with the error, so it is retried a day later like any other and never holds up the rest; the card then says the last attempt did not complete. **Records past the window: archive or remove.** With `AIOPS_ARCHIVE_BUCKET` set (chart `config.AIOPS_ARCHIVE_BUCKET`, region `AIOPS_ARCHIVE_REGION`), the select on the card defaults to **Archive, then remove**: each closed month of a workspace is written to the bucket as one JSONL+gzip object with its manifest (`tenants//archive//`), credentials masked, verified by SHA-256 after the upload, and only then removed in one transaction that writes a `retention.archived` audit row carrying both SHA-256s. A record edited between the export and the removal is held for the next batch. **Remove** keeps today's behaviour. Without a bucket the card offers **Remove** only ("Archive" says it needs the bucket) and records are removed as before. The archive runs as a detached job, one batch at a time on the control plane, with a one-hour timeout; an interrupted batch resumes from its state an hour later, and a month that failed three times in a row waits a week. It accepts only the control plane's own Pod Identity (or web-identity) role: credentials resolved from anything else — `AWS_ACCESS_KEY_ID` in the environment, a shared credentials file, the node role — refuse to run and name the method. The card shows the archive storage line (bucket, region and the credential method, or the problem), the newest batch, and the workspaces whose last attempt failed with the error. **Paused** means the archive was chosen or has already run but the bucket setting is gone: nothing is removed until the archive is fixed or you set the action to **Remove** — removing the bucket is a kill switch, never a silent fall-back to unarchived deletes. Watch `aiops_archive_last_success_timestamp_seconds{tenant}` (alert when it is more than two days old). The archive is not a database backup; restoring archived records into the database is not offered in this release (the manifests list every record, so it can be written later). Per workspace: `GET /api/aiops/{workspace}/archive/batches` (`members.manage`) lists the batches; `GET /api/aiops/{workspace}/archive/batches/{batch}/training.jsonl.gz` exports one archived month as fine-tuning data, fully PII-scrubbed with the workspace's salt and audited as `training_data.exported`. ### Licence The **Licence** card on this page shows the offline licence file the control plane verifies at boot and on every scheduler tick (an Ed25519 signature; no licence server is called). It reports **Customer**, **Expires**, **Features**, **Workspaces** (the count against the licence's ceiling, or *no ceiling*) and **Last verified**. The licence is installed through the Helm chart, `licence.value` or `licence.existingSecret` plus `licence.publicKey`; the card is read-only and neither installs nor renews anything. Read it before handing a deployment to a customer and after every renewal. | State on the card | What it means | |---|---| | **Valid** | The licence verifies and is inside its validity period. Every other run and execution guard still applies. | | **Expired — in grace** | The licence expired; the banner names the days of grace left. Install a current licence before the grace period ends. | | **Expired — agent runs paused** | Grace is over. New agent runs and approved executions are paused the way the workspace kill switch pauses them; alerts, incidents, audit and reports keep working. | | **Invalid — agent runs paused** | The installed file did not verify against the public key. Correct the installed licence or key before starting new work. | | `unlicensed` (the state badge) | No licence file or no vendor public key is installed; the card's note names which. Nothing is paused: agent runs and approved executions start as usual, and **Customer** and **Expires** read —. It is not evidence that the deployment is licensed. | After a deploy or a renewal, confirm the state and the **Last verified** time moved. This card is the read-out; the install, renewal and key-change procedure is the licence chapter of the Day-2 operations runbook that ships with the installation. ### Default email relay (SMTP) API: `/api/aiops/admin/smtp` (`GET`, and `PUT` with the relay fields). Requires platform superadmin. The **Default email relay (SMTP)** card holds the relay every workspace uses for email notifications until it configures its own under **Notifications & on-call → Email delivery** in the portal. Enter **Host**, **Port**, **Username**, **From address** and, when the relay needs one, **Password**; keep **STARTTLS (verified)** on for a relay that supports it, then select **Save relay**. The password is write-only: a blank password on a later save keeps the stored one (the label reads *set; leave blank to keep*), and the value is never returned. When the deployment has no secret-store key the card warns *AIOPS_TENANT_SECRET_KEY is unset — a password saved here is stored in PLAINTEXT in the database*; set the key before entering real credentials. **Clear** removes the saved platform relay. It does not necessarily stop email: the deployment's `AIOPS_SMTP_*` environment variables, when set, take over, and the card says so (*No DB relay set — email currently rides the AIOPS_SMTP_* environment variables*). With no workspace relay, no saved platform relay and no environment relay, email targets are skipped with a warning. A saved relay is configuration, not proof of delivery: send a test notification to a receiver you control and keep that receipt. ### Built-in detection (AWS) API: `/api/aiops/admin/builtin-detection` (`GET`, and `PATCH` with `enabled`). Requires platform superadmin. The **Built-in detection (AWS)** card is the platform freeze for the deterministic CloudWatch rules a workspace owner turns on under Datasources (read-only `ListMetrics` and `GetMetricData`, billed to the customer's own account). The switch **Built-in detection may run for workspaces that turned it on** stops evaluation for every workspace at once when it is off. It freezes rather than clears: nothing is evaluated and nothing is resolved, so detector alerts already firing stay open until the switch is back on, and then evaluation resumes for every workspace that enabled the feature. Use a workspace's own detection setting to change that one workspace. Do not read the platform freeze as a source clear, and read the saved switch together with the open alert states when verifying a freeze or its lift. Workspace guide: [Built-in detection](/help/builtin-detection.md). ## Customer connection setup Route: `/ops/platform#connection-setup`. Nav: **Platform defaults** under **Config**, the **Customer connection setup** section below the tier map. API: `/api/aiops/admin/connection-setup`. Requires platform superadmin. This section publishes the public installation details shown in customer setup panels: the private-agent image and chart, and the platform's existing AWS role/account and workload-identity issuer/subject. 1. Select **Edit connection setup**. 2. Enter the **Private agent image** as an existing registry/repository and version tag or digest. Customers still need registry authentication if the image is private. Add an actual HTTPS or OCI chart location only if you distribute a chart. 3. Fill in the platform AWS role/account and the workload-identity issuer/subject this deployment actually uses. Google Cloud keyless setup remains unavailable on this shared deployment; publishing runtime metadata does not enable it. Do not enter secrets here. 4. Select **Save connection setup**. The current values show whether they are saved overrides or inherited deployment defaults. Empty fields fall back to environment values. **Expected result:** customer setup panels display the published values. This does not install an agent, create federation or grant cloud permissions. Verify an actual customer agent heartbeat, datasource read and investigation before treating its setup as complete. Each tenant's AWS External ID is generated separately and cannot be changed here. ### Keyless AWS: any-account mode and prepared pairs {#keyless-aws-any-account-mode-and-prepared-pairs} **Any-account mode** is a deployment setting: `AIOPS_KEYLESS_AWS_ANY_ACCOUNT=true` (enabled in the customer example values). It requires both workload roles: the control plane's **Platform AWS role ARN** here or `AIOPS_PLATFORM_AWS_ROLE_ARN`, and the runner role configured as `aiops.runner.serviceAccount.awsRoleArn` in the chart. Their deployment permissions allow assuming estate roles named `aiops-*`. With complete deployment identities, an eligible workspace connects another AWS account without a per-account staff step: the guided setup creates `aiops--read` and `aiops--write`. The platform's own account still needs an exact staff-prepared pair. A workspace with a private-network agent or staff AWS setup remains on the prepared flow. Per-workspace preparation is API-only. Read `GET /api/aiops/admin/tenants/{id}/cloud-runtime`, then `PUT` that path while preserving unrelated fields. For the platform's own account, publish one `aws_targets` entry with the exact pair and leave `reader.aws_role_arn` empty; this pair may use any role names. Outside that account, prepared roles must be named `aiops-*` with no path, otherwise the API refuses them with 422. A private-agent workspace needs its own reader published with its prepared account pairs. Publishing metadata alone grants nothing; the deployment roles and customer trust/permissions must agree. **Who creates the pair for the platform's own account.** Platform staff (a superadmin; after handover, your own platform administrators) choose the two role names and publish them with the `PUT` below. The roles themselves are created by the workspace's guided setup: once the pair is published, **Keyless cloud access** in that workspace previews the CloudShell command for that account under the published names, and an AWS administrator of the account runs it there, as in [Connect AWS](/help/setup-aws.md). Until the pair is published, **Keyless cloud access** in that workspace keeps **Preview setup command** disabled for that account and shows **Platform setup required**: *The AWS account above hosts the platform itself. A platform administrator must prepare its exact read/write role pair before a command can be prepared.* Called directly, `POST /api/aiops/{tenant-id}/cloud-setup/preview` answers 409 *This AWS account hosts the platform itself: platform staff must prepare its exact read/write role pair first.* Read the workspace's current setup with an **admin-scope** personal access token of a superadmin account. `` is the last part of the workspace's `/ops/workspaces/` address. ```bash curl -H "Authorization: Bearer " \ https:///api/aiops/admin/tenants//cloud-runtime ``` A workspace in any-account mode with nothing prepared answers: ```json { "tenant_id": "", "configured": { "reader": {"aws_role_arn": "", "oidc_issuer": "", "oidc_subject": ""}, "aws_targets": [], "azure_bindings": [] }, "runtime": { "mode": "remote", "reader_source": "tenant_reader", "reader": {"aws_role_arn": "arn:aws:iam::123456789012:role/aiops-example-aiops-runner", "oidc_issuer": null, "oidc_subject": null}, "aws_targets": [], "azure_bindings": [], "aws_configured": true, "azure_configured": false, "verification_status": "not_verified", "aws_any_account": {"read_role_name": "aiops-acme-read", "write_role_name": "aiops-acme-write", "platform_accounts": ["123456789012"]} } } ``` `configured` is what staff saved, in the shape the `PUT` takes. `runtime` is what the workspace's setup reads: `reader.aws_role_arn` is the identity its reads run as (here the deployment's runner role); `aws_any_account` appears only in any-account mode, and its `platform_accounts` are the accounts that need a prepared pair; `verification_status` is always `not_verified`, because publishing proves no access. Send `configured` back with the pair added. The `PUT` replaces the whole setup, so keep every field you do not mean to change; `{}` clears it, approved Azure bindings included. ```bash curl -X PUT https:///api/aiops/admin/tenants//cloud-runtime \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "reader": {"aws_role_arn": "", "oidc_issuer": "", "oidc_subject": ""}, "aws_targets": [ {"account_id": "123456789012", "read_role_arn": "arn:aws:iam::123456789012:role/aiops-acme-read", "write_role_arn": "arn:aws:iam::123456789012:role/aiops-acme-write"} ], "azure_bindings": [] }' ``` It worked when the response, in the same shape as the `GET`, shows the entry under both `configured.aws_targets` and `runtime.aws_targets` and still carries `aws_any_account`. The change is audited as `tenant.cloud_runtime`. The API answers 422 for an unknown field, a second pair for the same account, a read and write ARN that are the same role or not in `account_id`, and a pair outside the control plane's own account whose names do not start `aiops-` with no path (`Prepared roles in AWS account must be named aiops-* with no path: …`); a token without admin scope gets 403. [Customer AWS prerequisites and setup flow](/help/setup-aws.md#before-you-start). ## Users Route: `/ops/users`. Nav: **Users** under **Platform**. API: `/users` (root-mounted, superadmin-only). Every account on this deployment, local and SSO. Tenant *membership* is a different thing and is granted inside each tenant, by that tenant's own administrator — see the tenant handbook's *People, access and audit* chapter. ![Every account on the deployment: local and SSO, active or disabled, with the superadmin badge that unlocks this console and nothing else.](/help/media/ops/ops-users.png?v=0.1.132) | Column | Meaning | |---|---| | **User** | Display name over the email. | | **State** | `active` or `disabled`, plus a `superadmin` badge. | | **Auth** | `local` or `oidc`, plus an `SSO linked` badge when the account is bound to an identity provider subject. | | **Last login** | Local date and time, or `never`. | The search box filters the loaded rows; any header sorts them. ### Create a platform user Requires platform superadmin. 1. Click **New user**. 2. Enter **Email**. It is lowercased on save and must look like an address; whitespace, quotes, angle brackets, `&` and backslashes are rejected with `invalid email`. 3. Enter **Password (min 8 chars)**. The minimum is enforced by the server, so a shorter one comes back as `password too short (min 8)`. 4. Optionally enter **Display name (optional)**. It defaults to the local part of the email. 5. Tick **Superadmin** only if this person is platform staff. 6. Click **Create user**. It worked when the dialog closes and the new row appears with `active` in **State**, `local` in **Auth**, and `never` under **Last login**. A duplicate address is refused with `email already exists`. API: `POST /users` (root-mounted; superadmin, and a personal access token needs the `admin` scope) with `{"email": "admin@example.com", "password": "", "display_name": "", "is_superadmin": false}`. The `201` response is the account, with the `id` that organization access takes as `user_id`. The account now exists but reaches no tenant data whatsoever. To give it access to a tenant, either have that tenant's administrator add it as a member from the portal, or take a support-access grant as described in the **Customers and workspaces** chapter. This screen is also the answer to the tenant-side dead end where an administrator tries to add a colleague who has no account yet: either their identity provider auto-provisions the account at first SSO login, or platform staff create it here. ### Disable or re-enable an account Requires platform superadmin. 1. Find the row and click **Disable**. 2. To reverse it, click **Enable** on the same row. It worked when the row's **State** flips to `disabled` (or back to `active`). Disabling deletes every active browser session for that user immediately, and blocks both session and API-token authentication from the next request. Existing API tokens are not deleted — they simply stop working while the account is disabled, and resume if you re-enable it. If the departure is permanent, revoke the tokens explicitly rather than relying on the disabled flag. One account resists this: the **bootstrap admin** named by `AIOPS_ADMIN_EMAIL`. Every control-plane start re-asserts its superadmin flag and re-enables it if it was disabled — a deliberate break-glass so an operator can never lock the deployment out — and with `AIOPS_ADMIN_FORCE_PASSWORD` set it also re-pins its password from the environment. To retire that account for good, remove those environment variables first, then disable it. ### Reset a password Requires platform superadmin. Only for an account with no workspace or organization seat (a support grant aside) that signs in with a local password: your own, another staff account, or one not yet added anywhere. Any other account is refused with `409 this account holds workspace access or signs in through SSO …` — a staff-chosen password would sign you in with that member's workspace authority and no support grant. Send such a local account **Email reset link** instead (below); a workspace owner can also reset a member's password with **Reset password** on the member's row in the portal. An SSO account is reset at its identity provider. 1. Click **Reset password** on the row. 2. Enter **New password (min 8 chars)**. 3. Click **Set password**. It worked when the dialog closes with no error. A reset also ends every browser session of the account, so anyone still holding one must sign in again with the new password. API tokens are separate credentials and keep working; revoke them explicitly when the reset follows a suspected compromise. ### Email a reset link For a local account with workspace or organization access that lost its password — a workspace's only owner, an organization administrator, a member of two workspaces. Requires platform superadmin and an email relay (the account's workspace relay, else **Default email relay (SMTP)**). 1. Click **Email reset link** on the row. 2. The page confirms `Reset link emailed to
.` The one-time link goes only to the account's own address and is never shown to you. It expires after the invite lifetime (seven days by default); the old password works until it is used, and using it sets the new password and ends every session of the account. Each workspace the account belongs to, or that its organization holds, records `member.password_reset_link` with `via: platform_staff` in its own audit log, so the customer sees that staff sent it; the platform log records `user.reset_link_sent`. It is refused with `409` for an SSO account, a disabled one, one with no workspace access (set its password directly) or when no relay is configured, and with `502` when the relay did not take the message. ### SSO-linked accounts Accounts that arrive through a tenant identity provider auto-provision at first login and appear here once they have logged in. You do not create them. They show `oidc` in **Auth** and an `SSO linked` badge, and usually have no local password at all. **Reset password** refuses an SSO account (409): a local password would be a second login path alongside SSO. Fix the account at the identity provider. An account created by SSO that is not platform staff never signs in with a password, even one set before resets were refused: the sign-in form answers `invalid credentials`. A staff account keeps its own password as break-glass. ## Billing {#billing} The **Billing** page (`/ops/billing`) is the platform-side statement of model spend: one UTC calendar month at a time, every tenant, grouped into the same five spend classes a workspace sees on its own **Usage & billing** page. The three surfaces — this page, a workspace's page and its organization view — are built from the same run records and never disagree about the money, only about how it is grouped. ![Billing: the platform's month as a statement — tiles, spend by class and the tenant list.](/help/media/ops/ops-billing.png?v=0.1.132) Top to bottom: 1. **Toolbar** — the month picker (UTC; **←** and **→** step a month, **This month** returns to the current one, titled *month to date* while it is still open), **Download CSV**, a search box on tenant name or slug, the **Hide zero-spend tenants** toggle (on by default — a tenant that ran nothing that month is noise in the list; switch it off when that absence is the question) and the sort: cost, highest first, or name. 2. **Tiles** — **Total spend**, the **Production** and **Lab** subtotals (a lab box's spend is booked on its own tenant and never folded into production), **Agent runs**, **Tokens**, and **BYO LLM tenants**: a count of tenants that bill their own gateway — their tokens and runs are here, their money is not on your invoice. 3. **Spend by class** — one stacked bar for the whole platform and a legend row per class: the customer-facing label (the technical class in the tooltip), its cost and its share. The classes: | Label | Class | Meaning | |---|---|---| | Investigations | `operating` | Every alert, pulled finding, scheduled or manual run and agent-API call: from the signal to the root-cause analysis and the change request. Incident and compliance alike. | | Validation | `verify` | The read-only check after a fix: did the condition clear, did the change hold. | | Estate x-ray | `xray` | Service discovery: the periodic scan that keeps the estate model and the knowledge current. | | Chat | `chat` | Ask the analyst. Fractional: it costs what people ask. | | Test Lab | `test` | Benchmarks and the lab box: deliberate test spend, never operating cost. | 4. **Tenants** — one row per tenant rather than a wide table: name, slug and badges (**BYO LLM**, **Lab**, **Disabled**), the month's total with its share of the platform, a mini stacked bar by class, and runs and tokens in small print — with `incl. $x estimated` when part of the month was priced from an estimate rather than a published or contract price. The row is a button: select it, or press Enter on it, to expand one cell per class — cost, runs, tokens — and an **Open workspace** link to `/ops/workspaces/`. Nothing on the page scrolls sideways, on a phone included. 5. **Download CSV** writes one line per tenant and class plus a grand-total line, with the class in its technical spelling, so a finance spreadsheet can join it to a workspace's own statement without translating labels. Its last columns: `estimated_usd`, the part of `cost_usd` priced from an estimate rather than a published or contract price, `unpriced_tokens`, the part of `tokens` no price was recorded for (shown at `$0` — resolve it before invoicing from the file), and `free_checks`, the validations that ran no agent ($0; `runs` counts agent runs only). Each is empty for a month sealed before it was recorded: unknown, not zero — such a month's `runs` still includes its free checks — and so is the grand-total cell when any tenant's is, never the sum of the tenants that know it. Above the tiles, an **Estimates** line appears while any model in use has no trusted price — *N models in use are booked from estimates; M are priced without a recorded source — fix on Platform defaults → Model price map*. A **Not priced — resolve before invoicing** line names each tenant with tokens booked on a model nobody has priced yet (a workspace endpoint model not mapped): those tokens are `$0` in the totals; map the models, then re-price that workspace's history. The tenant row repeats the count. Two things decide whether these numbers are real: - **The price card.** Cost is booked per call from the recorded prompt, completion, cached-read and cache-write tokens, priced by the entry for that model id on **Platform defaults → Model price map** that was in force when the run started: an official AWS price (Bedrock), the publisher's list price through a mapping (models.dev, every other model), a manual price with its source note, or the deployment's own map (`AIOPS_MODEL_PRICES`, source not recorded). A model with no entry is still booked — from the model library's own price table, or the `AIOPS_PRICE_PER_1M_*` fallback rates, so budgets and caps keep working — but labelled an **estimate** everywhere the money shows; only an id neither knows books **$0**. If a tenant's spend reads zero while their agents clearly run, or a model sits under *Needs attention*, press **Fetch list prices**, **Map** it to its publisher's model, or record its contract price. A card entry *without* `cache_read_per_1m` bills cached prompt reads (most of a run's prompt tokens) at the full input rate — an over-count; see [Cached prompt reads — price them explicitly](/help/ops-llm-gateway.md#cached-prompt-reads-price-them-explicitly) in the LLM gateway chapter. A corrected price applies from its date; to correct the past, date it back and run `POST /api/aiops/admin/rebook-costs` (dry run first). - **Test spend is kept apart**, under **Test Lab**: benchmark runs are operating cost of the *platform decision*, not of the tenant's incident response, and mixing them would make both numbers useless. The same rule keeps them out of every operating figure, the service delivery report and the model-tier suggestions. X-ray and compliance scans are billed once, on their own scan record. Deterministic layers (ingestion, correlation, routing, approvals) cost $0. ```mermaid flowchart LR I[LLM call] --> P{card entry in force
for the model id?} P -- AWS price list / manual --> T[booked, trusted] P -- deployment map --> U[booked, source not recorded] P -- none --> L{model library
or fallback rate?} L -- yes --> E[booked, labelled estimate] L -- no --> Z[booked at $0
fix the card] T --> R[Billing rollup
per tenant / model] U --> R E --> R ``` ## Admin MCP server {#admin-mcp-server} The same platform administration is available to your own coding agent over MCP, at `/mcp/admin`, for platform superadmins only. Its credential, tools and catalog-change rules are in [Admin MCP server](/help/ops-admin-mcp.md). ## Platform audit log Route: `/ops/audit`. Nav: **Audit log** under **Config**. API: `/api/aiops/admin/audit`, `/api/aiops/admin/audit/verify` and `/api/aiops/admin/audit/export`. Requires platform superadmin. Append-only and cross-tenant: every mutation, approval decision, model switch and agent action across every tenant. Every row is sealed within about a minute into a sha256 hash chain — one chain per tenant, plus the **platform chain** for rows with no tenant (catalog, platform-default and staff actions). ### Verify the chains Requires platform superadmin. 1. Leave the tenant dropdown on **All tenants** to check every chain, or choose a tenant to check only its chain. 2. Tick **Full replay** to replay every row from the first one instead of the newest 500 per chain. Replay time depends on chain size, and only one full replay runs per chain at a time. 3. Click **Verify chains** (**Verify chain** with a tenant selected). A table lists each chain with its result (`intact` or `broken`), the rows replayed, the first bad `seq`, the rows still awaiting their seal and the head (`seq` and the first 12 characters of its hash). `broken` means a sealed row of that chain was altered, removed or reordered after sealing — the application cannot do that, so it points at direct database access. Export that chain as evidence before anything else changes. ### Export a chain Requires platform superadmin. 1. Choose the tenant whose chain you want, or leave **All tenants** to export the platform chain. The button names which: **Export chain** or **Export platform chain**. 2. Click it, pick **From** and **To** (browser calendar days, **To** included), the **Format** (**NDJSON** re-verifies offline, **CSV** is for reading) and **Rows per file** (10,000 by default, at most 50,000). 3. Click **Download**. The file holds the period's sealed rows oldest-first with `seq`, `prev_hash` and `row_hash`. If the period holds more rows than one file, **Download the next file** continues from the last `seq`. The action filter and the prefix chips do not apply. The export is recorded **in the exported chain** as `audit.exported` with `via: platform-admin` — a tenant export lands in that tenant's own audit log, so the tenant sees that staff took its trail. `scripts/verify-audit-export.py` re-verifies an NDJSON file without access to the platform; the tenant handbook's *People, access and audit* chapter describes the algorithm and the cursor (`after_seq`) for scripted archives, and the written retention policy is [Audit trail retention policy](/help/ops-audit-retention.md). ### Filter the log ![The cross-tenant audit log: the tenant dropdown, the exact-action box and the action-family chips, which all search the whole log on the server, Verify chains and Export platform chain, then each event's time, action, target, actor, tenant and detail, newest first.](/help/media/ops/ops-audit.png?v=0.1.132) Requires platform superadmin. 1. Choose a tenant from the tenant dropdown, or leave it on **All tenants**. 2. Either type an **exact** action name in the action box — for example `approval.executed` — or click one of the prefix chips: `approval.*`, `model.*`, `xray.*`, `feature.*`, `runner_token.*`. 3. To reset, click **Clear**. The button only appears once at least one filter is set. It worked when the table redraws showing only matching rows, and the chip you clicked is highlighted. The tenant filter, the exact-action filter and the prefix chips are all applied by the server, over the whole log; the table shows the latest 100 matches. A chip is sent as `action_prefix` and matches every action that starts with it, so `approval.*` also finds approval rows older than the newest 100 events. Typing an exact action clears an active chip, and clicking a chip clears the exact action. `admin_audit` and its `audit_query` alias take the same `action` and `action_prefix` filters. To read past the latest 100 matches, page with `limit` and `offset` over the API. Columns are **When**, **Action**, **Target** (the affected object — for a catalog change the connector key, otherwise for example an incident, approval or scan id; `—` when the event names none), **Actor**, **Tenant** (the tenant's name, or `platform` for rows with no tenant) and **Detail** (the event payload as JSON, truncated to one line with no way to expand it). **Actor is an identifier, not an email.** It holds the acting user's **id**, or `agent:` for an agent action, or `system`. To turn an id into a person, look it up on the **Users** page. The platform-configuration actions from this chapter appear as `datasource_type.create`, `datasource_type.update`, `datasource_type.delete`, `platform.model_tiers_default` and `platform.auto_investigate_defaults` (its payload carries the defaults before and after), all with `platform` in the **Tenant** column. A workspace override set on the workspace page is recorded on that workspace's trail instead, as `guardrails.updated` with `via: platform_console`. Platform account changes appear the same way, each with the account's email in **Detail**: - `user.created` — the account was created; carries the superadmin flag. The actor is the superadmin who created it, or `system` for an account created at its first SSO login (`auth_source: oidc`, with the issuer that provisioned it) or by the bootstrap at startup (`via: bootstrap` — the `AIOPS_ADMIN_EMAIL` admin, or `admin@local` on an empty database with no bootstrap variables set). - `user.updated` — a superadmin changed it; carries only the fields that changed (`disabled`, `is_superadmin`, `display_name`) and `password_reset: true` for a password reset, never the password. Actor `system` with `via: oidc_superadmin_emails` is a login through the platform's own SSO issuer that promoted an address listed in `AIOPS_OIDC_SUPERADMIN_EMAILS`; it fires again at the first such login after a demotion. - `user.sso_linked` — actor `system`: an SSO login bound its identity to an existing account with the same email; carries the issuer. An account whose password was set through a workspace invite link is never bound this way, and so never promoted through `AIOPS_OIDC_SUPERADMIN_EMAILS`: the inviting owner held that link, so the address was never proven. The login is refused with `account was activated from an invite link — sign in with its password`, and a workspace invite cannot create an account for an address on that list. - `user.email_synced` — a tenant identity provider renamed the account's address; carries `from` and `to`. - `user.deleted` — a superadmin deleted it. A delete refused because the account authored recorded decisions writes nothing. - `user.password_changed` — the user changed their own password. - `user.reset_link_sent` — a superadmin emailed the account a one-time password reset link (**Email reset link**); carries `email_sent` and `expires_at`, never the link. - `user.bootstrap_reasserted` — actor `system`: a restart re-promoted, re-enabled or (with `AIOPS_ADMIN_FORCE_PASSWORD`) reset the `AIOPS_ADMIN_EMAIL` bootstrap admin. ### What the log will not tell you Two things worth knowing before you rely on this page as evidence: - **The table shows the latest 100 matches and one line of payload.** It does not page, and **Detail** cannot be expanded. `datasource_type.create`, `.update` and `.delete` write an empty payload, so their **Detail** reads `{}` and the **Target** column names the connector. `platform.model_tiers_default` is the exception: its payload carries the saved map, so tier changes are fully readable on screen. For more rows or the full payload, read the same rows over the API: ```bash curl -H "Authorization: Bearer " \ "https://aiops.example.com/api/aiops/admin/audit?action=datasource_type.update&limit=200" ``` The endpoint takes `tenant_id`, `action`, `action_prefix`, `limit` (default 100, capped at 500) and `offset`. The console sends `tenant_id` and either `action` or `action_prefix`. - **Current MCP catalog edits use the same audit names as REST:** `datasource_type.create`, `datasource_type.update` and `datasource_type.delete`. Older records from the retired MCP implementation may still use `.created` and `.updated`; include those historical names when reviewing older periods. ## Operational notes ### One control-plane replica The control plane ships with **one replica**. The cron scheduler that fires scheduled pillar agents, the sweep loops (audit seal, rollback sweep, licence check), boot recovery and the runner provisioner must each run in one process only. Leader election (chart `leaderElection.enabled`, off by default) keeps them on one elected replica through a PostgreSQL session advisory lock while every replica serves API and ingest traffic. Without it, a second replica double-fires every schedule — which a tenant experiences as duplicate compliance sweeps and duplicate executive reports, not as an error — so the chart refuses to render more than one replica unless leader election is on. The shipped Helm chart pins `replicaCount: 1` with the `Recreate` strategy because `/data` (skills and the per-tenant lakehouse) is a ReadWriteOnce volume that pins every replica to one node. Do not scale past one replica unless leader election is on, `/data` uses a ReadWriteMany class, and the database connection is direct — a transaction-mode connection pooler breaks the elector's session lock. Horizontal capacity for agent work comes from the **runner plane** — per-tenant runner processes — not from control-plane replicas. The runner plane is described in the **Customers and workspaces** chapter. ### A deploy is a short UI restart `Recreate` means the old pod stops before the new one starts, so the console, the portal and the API are briefly unavailable, typically for seconds. What that costs depends on the execution mode. In **`remote`** mode — the chart default, though the application default is `embedded` — investigations and estate x-ray scans run in per-tenant runner processes and are untouched. The control plane only enqueues, and the runners steal their own expired leases. Two things still live in the control plane and **are** lost: insights **chat** turns and **benchmark** runs in flight, which the next boot marks failed. Browser event streams reconnect and replay from the database after the restart. In **`embedded`** mode, in-flight investigations and scans are re-queued at boot with the attempt counter incremented, and resumed from their committed transcript — the transcript is written every step, so a resume replays at most the one lost step. Runs whose retries are exhausted, and runs that are not resumable by nature (chat, benchmark and agent-API turns), are failed and the incident escalated to a human. An in-flight recheck is closed out with an `uncertain` verdict rather than left claiming an agent is still verifying. Incidents parked at the human approval gate are never affected in either mode. Those rows are left alone and resume when someone decides. Every start also performs an **additive-only** schema migration — new tables and columns, nothing dropped or rewritten — then re-asserts the bootstrap admin, re-syncs the builtin connector catalog, the builtin RBAC roles and the builtin skill pack from code, hydrates the platform tier cache, and rebuilds the on-disk skill cache. ### Verify a release {#verify-a-release} Requires platform superadmin for steps 2 to 6. 1. Check that the unauthenticated health endpoint `GET /healthz` returns `{"ok": true}`. 2. Sign in to `/ops`. Confirm that **Customers** lists actual organizations and **Workspaces** lists the expected Tenant rows. Overview workspace totals include labs; Test Labs is a subset, and **Unassigned** reflects the direct organization link. A matching name or **Lab of** relationship must not invent a customer assignment. 3. On **Overview**, check the **Agent runners** panel: the exec mode and provisioner in the card description, and a `runner live` badge for each active tenant. `lease expired` on a tenant with running work means its runner has not checked in; a red `N retries exhausted` means that many rows have burned every attempt and need a look. The panel is documented in full in the **Customers and workspaces** chapter. 4. Open **Connector catalog** and confirm the expected types are listed. This proves the catalog re-sync ran. 5. Open **Platform defaults** and confirm your tier map is shown rather than the environment-variable notice. This proves the `PlatformSetting` row survived the deploy. Cache hydration itself is only visible in the startup log. 6. Open **Audit log** and confirm fresh rows are arriving. 7. Finish with a tenant-side smoke test in the portal: an incident opens, an investigation runs, and a remediation plan lands for a human. It worked when all seven pass. If step 3 shows `no claims` for every tenant in `remote` mode, no runner is deployed — that is the cause of the tenant-visible symptom "my investigation has been queued for ever". ![Release verification: customer and workspace totals are separate, while runner claims show activity without revealing workspace content.](/help/media/ops/ops-overview.png?v=0.1.132) ## Use cases ### Platform owner: onboarding an in-house monitoring system Your organization runs an internal alerting service that no vendor connector covers. Create a custom type with key `example-alerts`, kind `api`, a `fields_schema` of `url` plus a secret `api_key`, and a `config_template` pointing `openapi_path` at its schema so the agent discovers its own endpoints. Set pillars `incident, secops`, and save it enabled. Every tenant can now connect it from **Connect datasource** with no code change and no deploy. Two things to tell them. A custom key lands in the `other` category and does not yet count toward pillar coverage. And an unknown key gets the classifier's default-deny rule, so only `GET` and `HEAD` calls are reads until its paths are known — safe, but it means every other verb arrives as a remediation-plan item rather than as a finding. ### Platform owner: swapping the gateway's model line-up The gateway retires a model id. Open **Platform defaults**, repoint the affected tiers, and click **Save platform defaults**. Tenants that never set their own map pick it up on their next job, on the control plane and on runners from 0.1.117 alike. A runner image older than that keeps the retired id until you update `AIOPS_MODEL_TIERS_DEFAULT` on it and roll it (see *How your default reaches a tenant's agent runs*). Tenants that did set their own map are unaffected by this page entirely — chase those from their tenant pages. ### Compliance officer: evidencing a configuration change A quarterly review asks who changed a connector definition. Query the audit API for both spellings, `datasource_type.update` and `datasource_type.updated`, read the `at`, `actor` and `target` of each row, then resolve each `actor` id to an account on **Users**. Pair it with the tenant-scoped audit described in the tenant handbook's *People, access and audit* chapter for the tenant-side half of the story. If the review also asks who created, promoted, disabled or deleted an account, query `user.created`, `user.updated` and `user.deleted` the same way; a `user.bootstrap_reasserted` row shows a restart that undid a disable or demotion of the bootstrap admin, and a `user.sso_linked` row shows when an SSO identity gained an existing account. ### SRE on-call: a connector suddenly has no tools An investigation stops reaching the metrics backend across several tenants at once. Check **Connector catalog** first: a type disabled during a vendor incident removes its tools from every tenant's next run while leaving the instances and their secrets intact. Re-enable it — `PATCH … {"enabled": true}` for a builtin — and the next run has them back. Nothing needs re-entering. ## Troubleshooting This table covers the platform console only. Tenant-visible symptoms are covered by the tenant portal handbook, which staff cannot reach from inside this console. | Symptom | Cause | What to do | |---|---|---| | Save rejected with "`` is a builtin, code-owned type — … would be reverted on the next deploy" | You edited a structural field on a shipped type. | Clone it to a custom key and disable the original. Structural fields on builtins are owned by the code. | | Delete rejected with "tenant datasources of this type exist; disable it instead" | At least one tenant, possibly not one you can see, has an instance. | Disable it instead. Delete only once every tenant has removed its instances. | | A deleted builtin reappears after a deploy, with its flags reset | Startup re-creates every code-owned type with the shipped defaults. | Disable builtins rather than deleting them. | | "Not valid JSON: …", "Must be a JSON array of {name, label, secret}." or "Must be a JSON object." under **fields_schema (JSON)** / **config_template (JSON)** | Browser-side validation before the request is sent. The message sits on the box it is about. | Fix the JSON in that textarea. `fields_schema` is an array, `config_template` an object. | | "Required." under **Key** or **Name** | Browser-side validation. | Both must be non-empty after trimming. | | A JSON error naming `string_pattern_mismatch` on `key`, or `string_too_short` on `name` | Server-side validation: a key must be lowercase letters, digits, `_` and `-`, start with a letter or digit and be at most 64 characters; a name must not be blank. | Use a lowercase key, for example `my-grafana`, and a non-blank name. | | "datasource type '``' already exists" | The key is taken, possibly by a disabled type. Keys are case-sensitive. | Search the table for the key, then edit or re-enable the existing row. | | A tenant cannot see a type in **Connect datasource** | The type is disabled. | Re-enable it in the catalog. | | A connected datasource produces no tools | The type is disabled, the instance is disabled, the type's **Pillars** list excludes the running pillar, the instance's own pillar allowlist excludes it, the instance carries a write-role credential, or an `api` config has no `url` or `base_url`. | Check **State** and **Pillars** here; ask the tenant to check the instance's role, allowlist and URL in the portal. | | A tenant reports that a kubeconfig or PEM key is rejected on save | The field declares `format: yaml` or `format: pem` and the pasted value lost its line breaks. | Correct behavior. Tell them to paste the full file into the multi-line box. | | Model dropdown shows only **Custom model id…** | No gateway configured, or its `/models` call failed. | Type the model id by hand; the page still saves. Diagnose the gateway with the **Connecting the LLM gateway** chapter. | | Tier map saved here, but agents still run on one model | A runner image older than 0.1.117 in `remote` exec mode — it never receives this page's map. | Upgrade the runner (or agent chart) image. Until then set `AIOPS_MODEL_TIERS_DEFAULT` on that runner, or give the tenant its own tier map. | | Platform defaults page shows the environment-variable notice after you saved | You are looking at a different deployment, the save errored, or every tier was later cleared. Clearing all four hands routing back to `AIOPS_MODEL_TIERS_DEFAULT`, and the notice then shows the map in effect. | Re-save and confirm the green "applies to every tenant" message, or keep the tiers cleared if the environment map is the one you want. | | Tier escalation never fires for any tenant | Only one distinct model is configured, so the ladder has no steps. | Configure at least two distinct tier models. | | "cannot demote/disable the last superadmin" | The lockout guard, which is **self-only**: you tried to disable or demote your **own** account and no other enabled superadmin exists. | Create a second superadmin first. | | A disabled account is enabled again after a restart | It is the `AIOPS_ADMIN_EMAIL` bootstrap admin, re-enabled by design; the restart writes `user.bootstrap_reasserted`. | Clear the bootstrap environment variables before retiring that account. | | "invalid email", "email already exists" or "password too short (min 8)" | Server-side validation on user create or patch. | Fix the field. Passwords are 8 characters minimum; addresses are lowercased and must not contain quotes, angle brackets or whitespace. | | A disabled user's API token still appears to exist | Disable removes sessions, not tokens. | Tokens are refused while the account is disabled. Revoke them explicitly if the departure is permanent. | | An audit event you expect is missing | The table shows only the latest 100 matches of the filters you set. Historical MCP records may use older action names; account changes made before the upgrade that introduced the `user.*` actions were never logged. | Narrow it with the tenant filter, an exact action or a prefix chip; over the API, page with `limit` and `offset`. Include legacy spellings when examining older records. | | **Verify chains** shows a chain as `broken` | A sealed row of that chain was altered, removed or reordered outside the application. | Export that chain at once as evidence, then review database access; rows before the first bad `seq` still verify. | | "a full replay of this chain is already running — retry when it finishes" | Another full replay of the same chain is in progress. | Wait for it, or untick **Full replay** for the 500-row check. | | Scheduled agents fire twice for a tenant | More than one control-plane replica without leader election. | Scale the control-plane deployment back to 1, or turn on `leaderElection.enabled`. | | Chat turns and benchmark runs failed after a deploy, investigations did not | Expected. Those two are control-plane-side and not resumable. | Re-run them. Nothing is lost but the run. | ## Limits and gotchas {#limits-and-gotchas} - **The connector key is permanent.** REST and the admin MCP create operation accept only lowercase letters, digits, `_` and `-` (starting with a letter or digit, at most 64 characters) and refuse anything else; neither changes the case for you. Choose the key carefully: there is no rename operation. - **Categories are not editable.** They come from a fixed map keyed on the connector key, which drives the tenant connect grouping and the pillar-coverage matrix. One key can carry several categories. A custom key resolves to `other` and does not close a coverage gap. - **Builtin structural fields are re-synced on every deploy.** That is a feature — a newly shipped field reaches existing databases with no migration — and it is the reason the API blocks those edits, and the reason deleting a builtin is undone at the next start. - **The same catalog CRUD is reachable over the admin MCP server**, through the canonical create, patch and delete handlers with their existing validation, builtin-field guard, permissions and audit names. - **Editing a type never rewrites tenant configuration.** Changes reach the next agent run through template rendering. A `fields_schema` change does not retro-fill existing instances; new placeholders stay unrendered until a tenant re-saves the datasource. - **The type description is not the tool description.** It is tenant-facing in the connect picker and reaches agents only through the estate x-ray `platform_context` catalog section. The agent's per-tool "Operator notes" line comes from the instance's description and is truncated to 400 characters. - **Tier maps are never merged.** A tenant that sets a single tier stops inheriting the platform default entirely, for all four tiers. - **Runners from 0.1.117 follow the platform tier default.** A db-mode runner re-reads it each time it polls, an api-mode runner the platform provisioned receives it with every job, and the provisioner copies `AIOPS_MODEL_TIERS_DEFAULT` onto the db-mode runners it creates. A private-network agent the customer runs does not receive it, and older runner images do not follow it (see *How your default reaches a tenant's agent runs*). - **Clearing every tier hands routing back to `AIOPS_MODEL_TIERS_DEFAULT`**, on the control plane and on the runners that follow this page. A tenant with no tier map of its own then runs on that map, else on `AIOPS_MODEL`. - **The console has no promote or demote control, and no delete-user control.** The superadmin flag is set at creation; changing it afterwards, or deleting an account, is an API operation (`PATCH` or `DELETE /users/{id}`). You cannot delete your own account. Prefer disabling to deleting, because deletion also removes the account's sessions and API tokens permanently. - **Account changes are audited only from the upgrade that introduced the `user.*` actions.** Creations, promotions, disables, deletions and password changes made before it left no row. - **The audit table has no date-range filter and no payload expander.** It returns the newest events first, 100 per request in the console and 500 maximum over the API. For a date range, export the chain (**Export chain** / **Export platform chain**, or `GET /api/aiops/admin/audit/export` with `since` and `until`): the file carries every column, `target` included. - **`Default readonly` is a label, not a gate.** The intrinsic write floor in the tool layer decides read versus write per call, and estate x-ray additionally refuses a gated call outright rather than pausing it. You cannot switch either off from this console, and no tenant can switch either off from the portal. ## Next steps - Next: [Connecting the LLM gateway](/help/ops-llm-gateway.md) - [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt) - [Documentation home](/help/)