# Release notes 0.1.156

> 27 October 2026: Four security hardening waves from the internal review — identity, the private agents and runners, change integrity, and the connector tools — and three fixes from the live run of…

_Release notes · Updated 2026-10-28 · Describes release 0.1.164_

## 0.1.156 — 27 October 2026 {#0-1-156-27-october-2026}

Four security hardening waves from the internal review — identity, the private agents and runners, change integrity, and the connector tools — and three fixes from the live run of 0.1.155. Two changes break an integration until an operator acts: a ServiceNow or Jira change ticket accounts for a captured write only when it arrives with the new **ITSM webhook token**, and a self-managed cluster's audit webhook needs its own per-cluster token (see the [upgrade row](/help/ops-upgrade-and-verify.md)). New routes: `POST /api/aiops/{tenant_id}/itsm-webhook-token`, `POST /api/aiops/{tenant_id}/k8s-audit-token/{cluster}`, `POST /users/{user_id}/reset-link`, `POST /api/aiops/{tenant_id}/changes/stats` and `POST /api/aiops/{tenant_id}/benchmarks/model-report`. New settings: `AIOPS_METRICS_TOKEN` (the chart sets it) and `AIOPS_KUBECTL_IN_CLUSTER` (set only by the private agent chart's `inClusterRbac` mode). One additive column, `runnertoken.serves_chat`, is created at start. No switch default changes.

### Security (identity) {#security-identity}

- **Issuer answers are bounded.** Every request to an identity provider — the bearer check's discovery, JWKS and userinfo, the sign-in's discovery, the callback's token and userinfo calls, logout's discovery and the IdP **Test** button — runs under a 10-second overall deadline (15 seconds for each sign-in exchange) and a 256 KiB cap counted on the wire. The platform asks for uncompressed answers and refuses a compressed one instead of inflating it. A signing key is checked by size before any arithmetic: a modulus of at most 4096 bits and an odd public exponent from 3 to 65537; anything else fails the signature. A discovery or JWKS fetch that fails falls back to userinfo as before.
- **The database connection is released around identity-provider calls.** The bearer check, `/oidc/login`, `/oidc/callback`, logout and `POST …/idp/test` end their read transaction before calling the issuer, so a slow or trickling issuer that a workspace chose no longer holds one of the control plane's pooled connections for every other workspace.
- **Keyless AWS with a placeholder key.** A `role_arn` connection whose key field holds a `${field}` template placeholder is keyless to the binder as it already was to the credential broker: the placeholder is dropped and the platform-minted External ID bound, audited as before. A row saved earlier that still holds a placeholder key fails closed until it is saved again, and the connection list reads its auth kind with the same test.
- **Keyless Google Cloud no longer uses the platform's identity.** A Google Cloud connection saved without a service-account key used the metadata server, the control plane's or the runner's own identity, shared by every workspace. The broker now refuses a missing key (`keyless Google Cloud is not supported`), the catalog marks the key required, and connections saved without one fail closed.
- **Invite adoption.** An account whose password was set through an invite link minted outside the workspace's organization is refused by another organization's member, group-admin and platform-console seat paths (`409`), never auto-links to the platform's own single sign-on, and is never promoted to platform staff; an invite cannot create an account for an address the platform promotes to staff (`400`). The issuer of a workspace identity provider is compared with its scheme lowercased, in the foreign-account check and in `sso_trusted_issuers`.
- **Staff no longer set passwords on workspace accounts.** A platform administrator's password reset is refused (`409`) for an account that holds a workspace membership or an organization seat, or that was born from single sign-on. In its place: a workspace owner's **Reset password** on a member's row mints a one-time link for a local member who belongs to this workspace alone, emailed to the member only when a relay exists (else shown to copy), audited as `member.password_reset_link`; and staff can email a reset link to the account's own address (`POST /users/{user_id}/reset-link`, platform administrators with an `admin`-scope token; the link is never shown to staff and the call is refused without a relay). Documented limit: a workspace's only owner, an organization administrator or a member of two workspaces can be reset only through the staff-emailed link, never by a staff-chosen password. A second limit stays: an organization's first administrator is seated by staff (`PUT /api/aiops/admin/groups/{group_id}/members/{user_id}`), and a local account staff created under **Users** keeps the password staff chose until the person changes it or a reset link replaces it — so that seat is now recorded in the audit chain of every workspace of the organization (`group.member_set`, `via: platform_staff`) as well as in the platform log.
- **Sessions end with the password.** A self-service password change keeps only the session that made it; a staff reset, a reset link and the bootstrap administrator's forced password (`AIOPS_ADMIN_FORCE_PASSWORD`) end every session. Personal access tokens are separate credentials and are unchanged. An account created by single sign-on that is not platform staff signs in through its identity provider only.
- **Admin scope on more routes.** A personal access token needs the `admin` scope to seat or re-role members, to install or change the workspace identity provider, to staff a workspace as a group administrator, and for the three token mints; reads and `POST …/idp/test` are unchanged.
- **The compliance mute list is protected.** `aiops_compliance_muted` in a workspace settings patch answers `400`; only `POST …/incidents/muted-rules/unmute` writes it, and it only removes entries.

### Security (private agents and runners) {#security-private-agents-and-runners}

- **Approval receipts name the run's own approved requests.** A private agent's `approval.*` audit rows are kept only for an approval of the run it checkpoints that a person approved; every other runner audit action is limited to the actions the engine emits, and the quiet-expiry sweep reads only its own platform marker.
- **Re-pointing a write credential is a release decision.** Changing a write connection to another role, or the config of a write connection that holds a stored secret (its **Test** and discovery send that secret to the configured endpoint), needs `approvals.decide` as well as `datasources.manage`. Echoing the unchanged config, editing other fields and completing a write connection with no secret yet stay open to `datasources.manage`.
- **Only an owner-designated agent answers workspace chat.** An agent token serves members' private chat turns only when it was created with **Answers workspace chat** (`members.manage`); only such a token makes the workspace's chat private or claims a turn. Tokens that had already connected before the upgrade keep serving chat; the designation cannot be removed from an existing token — revoke it and create a replacement.
- **The demo webhook token opens nothing with the demo seed off.** With `AIOPS_SEED_DEMO` off the published demo token is refused on every workspace, including one seeded while the seed was on.
- **Lab webhook token slots.** The Test Lab's reused webhook token lives under a per-workspace secret, so one workspace's provision no longer overwrites another's; the old shared slot is migrated once and removed.
- **A lab box closes with its parent.** A Test Lab box whose parent workspace is deactivated or gone is closed to its webhook token, its runner tokens, run starts and the runner provisioner, and skipped by every scheduler tick and sweep (crons, verifies, detection polls, synthetic checks, harvests, validation reads, rollbacks); it reopens when the parent is activated. A deactivated workspace itself is skipped by the automatic rollback sweep and the lakehouse collector as well: its failed changes wait for **Activate** instead of being rolled back automatically.

### Security (change) {#security-change}

- **ITSM ticket approvals are bound to an ITSM credential.** Only a ServiceNow or Jira change ticket posted with the workspace's ITSM webhook token (`POST /api/aiops/{tenant_id}/itsm-webhook-token`, `members.manage`, MCP `mint_itsm_webhook_token`; shown once, hash stored) stores its ticket fields and is stamped `itsm_verified`; only a re-send from the same ITSM refreshes them, and it dedupes only onto that ITSM's own ticket row. The ticket scan and the ITSM match take verified rows only, so a ticket from the generic change feed, the shared webhook token or an earlier release never covers a write and never crowds out one that does. BREAKING CHANGE: with aiops_unauthorized_change_incidents on, a ServiceNow or Jira change ticket sent with the shared webhook token no longer accounts for a captured write. Mint the ITSM webhook token (POST /api/aiops/{workspace}/itsm-webhook-token, members.manage; an admin-scope token through the API or MCP) and send change tickets with it. Until then the writes those tickets covered are flagged; a ticket recorded earlier counts again on its next re-send with the token.
- **A per-cluster Kubernetes audit token.** The audit webhook `/webhook/kubernetes-audit/<slug>/<token>/<cluster>` accepts only that cluster's own token (`POST /api/aiops/{tenant_id}/k8s-audit-token/{cluster}`, `pillars.manage`, MCP `mint_k8s_audit_token`; shown once, hash stored, minting again rotates it), never the shared webhook token; the form without the cluster segment answers `404`. What the API server posts is recorded as the platform's own audit trail, which a token every integration holds could forge: a benign write posted ahead of the API server made the genuine delivery, a refused write included, a duplicate. BREAKING CHANGE: a self-managed cluster's audit webhook posting with the workspace webhook token, or to /webhook/kubernetes-audit/<slug> with ?cluster=, is refused (401 / 404) and its writes are not recorded. Mint the cluster's token (POST /api/aiops/{workspace}/k8s-audit-token/
  <cluster-name>), put it in the audit webhook kubeconfig's token: and
  restart the kube-apiserver.
- **The freeze window is judged at the provider's time.** A write an audit log captured (CloudTrail, the Azure activity log, the Kubernetes API server) is judged at the time the log recorded it, so a write made during the freeze is flagged when a later pull reads it and one made before it is not flagged by a pull during it. A pushed, synced or ticketed change is judged when it arrived: a commit dated inside a freeze but pushed after it is not flagged, and one dated before it never clears a push received during it.
- **An L3 quorum counts per action and per current approver.** An in-run L3 release counts a vote only for the command it released; an approver's edit waits on the request and the quorum restarts whenever a release names a different command, so only a command two people saw runs. The pending card shows the proposed edit and releases it. An earlier vote counts only while that member may still release the tier — a removed, disabled or re-roled approver no longer completes a quorum, at the next release and at the dispatch reservation.
- **A platform change's blast-radius list stays out of the fold.** A platform change records its incident's impacted services; the change-window rule and the dependency pin read them as the services the change touched and folded every new alert on a dependent into the fix's incident. Only a captured change's `services` pin or fold now; a platform change matches on its own service and resource.
- **The ledger trusts its own capture.** A change-feed sender cannot swallow the platform's own capture of a write by posting its key first (the trusted capture dedupes only against rows the platform captured); an RDS parameter-group write takes its before-value from harvested writes only, never from a reported row; a revert operation (an ingress rule, an NSG rule, a restore of S3 Block Public Access) is compiled only from a revert drafted from the ledger, never from a plan sent in; and the rollback sweep skips decided rows before its limit, so escalated failures in one workspace no longer hold the window for a new failure elsewhere.
- **A validation window reads the alarm at the source.** A post-change round derived from a CloudWatch alert builds its rule from the alarm as CloudWatch holds it (`DescribeAlarms` through the read connection), not from the notification body. An alarm CloudWatch does not confirm by name and ARN, or whose dimensions or description hint differ from the body's, is a neighbour: it can flag a round, never decide one. A body naming any metric with an always-breached threshold could fail every round, mark a working fix as still present and arm the rollback sweep.
- **Credentials stay out of approval records.** `approval.requested`, `approval.executed` and `approval.edit_rejected` store the gated call's invocation as the approval page shows it, with any credential hidden, at every producer (the engine, the database persistence and a private agent's checkpoint, which no longer relies on the agent image); the audit log and the incident trail hide them in rows written earlier, and the export keeps stored rows as they are so the chain still verifies. The decision response (`POST …/investigations/{id}/decisions`, `aiopsctl` and the MCP tool) answers the approval queue's own projection.
- **The Test Lab's LKG replay restores only what the lane overlaid**: the lane's own workspace keys, the incident pillar gate and `config_history` on an AWS read connection, never an arbitrary settings key, connection field or pillar row a member wrote into the ledger; a `topology` record removes only the lane's own declared edge (services named `aiops-lkg-…-<nonce>`), never another edge of the workspace's graph.

### Security (connector tools) {#security-connector-tools}

- **A kubeconfig is a static identity.** A kubeconfig whose user runs a credential plugin (`exec`, such as `aws eks get-token` or kubelogin, or the older `auth-provider`), or that names a file on the platform host (`tokenFile`, `client-certificate`, `client-key`, `certificate-authority` as a path), is refused at save (`400`) and at every use — tool calls, **Test** and the inventory reader answer the same message — however the key is spelled (YAML tags, escapes, a file PyYAML cannot read). Accepted are a static or service-account `token`, `client-certificate-data` with `client-key-data`, and `certificate-authority-data`. Every connector process also runs without the pod's in-cluster API discovery, so a kubeconfig that resolves to no server fails instead of answering with the platform's own service account; a `{file:…}` left by a blank secret is a refusal too. The private agent chart's `inClusterRbac` mode is the one opt-in (`AIOPS_KUBECTL_IN_CLUSTER=1`, set by the chart).
- **Connection Test runs the catalog's probe only.** A generic command-line connection's **Test** runs the catalog's probe arguments, never operator-supplied `probe_args`, and never hands an argument list to an interpreter binary; the ssh template puts `--` before the destination so a user field can no longer be read as an option (existing rows pick the template up at the next start).
- **Answers from workspace-chosen endpoints are capped and time-boxed.** The Test probe, `http_get`, `web_fetch`, every `<name>_request` tool, the Prometheus tool, the trace sync, a validation round's Prometheus read, the firewall inventory, synthetic checks, the lakehouse collector, the MCP clients, the ServiceNow, Jira and Azure DevOps ticket clients, the GitOps pull-request providers (GitHub, GitLab, Azure DevOps, Gitea) and a connection's OpenAPI discovery stream their answer under a total deadline, inflate gzip and deflate with a bounded output, refuse other encodings and stop at 2 MiB (a collector pull past 16 MiB is an error status); a notification target's answer body is never read. About 64 KiB of gzip could grow to 64 MiB in the control plane before.
- **Per-workspace shares of shared slots.** A synthetic check takes one of its workspace's three probe slots before one of the ten global ones; a workspace holds at most one of the two PDF extraction slots; each workspace's lakehouse collection runs in its own detached task (at most four pulls at once, 120 seconds a specification), no longer inside the leader tick.
- **PDF extraction in a bounded child.** An uploaded PDF is extracted in a child process with a 192 MiB address-space and CPU limit, stopped at the 20-second deadline; the pages it finished are kept and `413 PDF too expensive to extract` means not even the first page finished.
- **A compiled step is sent as reviewed.** A step compiled from a reviewed plan or a step runbook — an Azure Run Command script included — runs with its own bytes; a `$` in it such as `MSSQL$SQLEXPRESS` is no longer filled from the incident after review. Hand-written ticket and webhook routes still fill their placeholders.
- **Reads that were writes, and views that were wider.** `GET …/changes/stats?export=true` and `GET …/benchmarks/model-report?export=true` saved an artifact; each export is now `POST` at the same path (**Export report** posts; MCP `export_change_stats`, `export_benchmark_model_report`) and the `GET`s ignore `export`. `GET …/compliance/discovery-findings` also needs `datasources.view` and an x-ray workspace, like the scan it reads from (no builtin role changes). The insights chat's incident evidence, incident detail and pending-approval tools answer from the same projections the REST readers use, so a credential in a gated command never reaches the chat model. A golden-set CSV cell starting with `=`, `+`, `-`, `@`, a tab or a carriage return is prefixed with an apostrophe, which the uploader strips; the risk register export's CSV block gets the same apostrophe (a scanner names the finding).
- **The metrics listener takes a bearer token.** With `AIOPS_METRICS_TOKEN` set, port 9464 answers only `Authorization: Bearer <token>` and `401` to anything else; the chart sets it (`secrets.AIOPS_METRICS_TOKEN`, else a stable value derived from `AIOPS_SECRET_KEY`), the ServiceMonitor presents it and runner pods get the same token from the control plane. The cluster network was not enough: the control plane dials workspace connectors from inside it. See the upgrade row for the scrape-job order.

### Fixes from the 0.1.155 live run {#fixes-from-the-0-1-155-live-run}

- **A refused rollback can be retried.** A rollback whose one call the provider refused before acting (a `4xx` other than `408`/`409`: a write role without the permission, a validation error), or that stopped before dispatch, locked the change for good. It now gives its reservation back: the attempt is kept (`rollback_attempts`, its audit row, the change page's list) and a person may release **Roll back** again under the same tier gate; the rollback sweep never replays it. The change page lists each refused attempt by its first line and the provider's error code (`HTTP 403 (UnauthorizedOperation)`); the whole answer stays in the record. An uncertain or partial outcome keeps its lock and the reconcile path.
- **The platform's own Lambda write is matched by function ARN.** CloudTrail names a function by its bare name on `UpdateFunctionConfiguration` and by ARN on `PublishVersion` and `UpdateAlias`, so the platform's own runtime write matched no receipt and was flagged as a change nothing accounts for. The harvest now records one identity for all three, built from the event's region and account; a function no step named, or an outside identity, stays flagged.
- **Patch Manager's last operation reads as a time.** The *not needed* reason printed the raw epoch (`last operation 1792992053.0`); it now reads `YYYY-MM-DD HH:MM UTC`.


## Next steps

- Next: [Release notes 0.1.155](/help/release-notes-0-1-155.md)
- [All What's new pages](/help/what-s-new/llms.txt)
- [Documentation home](/help/)