# Upgrade and release verification

> The deployment pipeline is the normal upgrade path.

_How-to · Updated 2026-10-28 · Describes release 0.1.164_

Evidence reviewed **8 October 2026**; operating baseline **0.1.130** (the 16 October 2026 handover build; see the [release notes](/help/release-notes.md)).

The deployment pipeline is the normal upgrade path. The current operating baseline is **0.1.130** (image tag `0.1.130`, chart `charts/aiops:0.1.130`, tag `v0.1.130`); select an image and chart from a published release record, not from an assumed newest tag.

## Prepare and approve {#prepare-and-approve}

1. Read [Release notes](/help/release-notes.md), including policy, schema and key changes. Book a maintenance window: the single control-plane replica uses `Recreate`, with about a minute of planned downtime.
2. Merge the released tag into `customer/<slug>` or rebase that branch on the release. The pipeline runs the files on that branch; stale files mean stale deployment steps. Historical runbook §3 example below; select the approved target from its release manifest before running it:

   ```bash
   git merge v0.1.130
   ```

3. Queue `deploy-customer.yml` from that customer branch with `customer` set to the installation slug, `imageTag` set to the approved tag, `chartVersion` set to the matching chart and `bootstrapDb=false`.
4. Read every unit in Plan before approving `aiops-<slug>`. An application-only upgrade normally shows **No changes**, with at most output-only changes. Adopting the any-account IAM policy below intentionally changes IAM; review that change. Unexpected infrastructure changes need review before approval.
5. The Deploy stage applies infrastructure, resolves and records the image digest and chart, checks signing when enabled, takes and waits for an RDS snapshot, and performs `helm upgrade --install --atomic --wait`.

### Adopt any-account keyless AWS on an existing install {#adopt-any-account-keyless-aws-on-an-existing-install}

Merging a release does not change the files render produced: `envs/<slug>/` and `deploy/customers/<slug>/values.yaml`. An installation rendered before 0.1.109 retains its broad estate-role assumption policy and has no `AIOPS_KEYLESS_AWS_ANY_ACCOUNT` flag in its rendered values.

Before adopting this mode, confirm every keyless estate role outside the platform account is named `aiops-*` without an IAM path; other targets become unreachable after the policy is narrowed. Bring `envs/<slug>/iam/tf/main.tf`'s `assume_estate` policy up to the released customer template, and add `AIOPS_KEYLESS_AWS_ANY_ACCOUNT: "true"` to the rendered values. Review the expected IAM policy change in that upgrade's Plan. The platform's own account still uses a staff-prepared pair. This is an upgrade procedure, not a claim that the rehearsal has adopted the new mode.

The customer pipeline runs a fresh apply after approval; it does not bind approval to a saved plan. Keep the branch stable through the run and review the final apply log. This accepted limitation is recorded as AR-19 in the Security review.

## Post-upgrade checks by release {#post-upgrade-checks-by-release}

Read every release between the installed version and the target. The release notes are the authority; this table lists the operator follow-up each release asks for and the evidence to keep with the release record. Three versions are never installed: **0.1.106** (void: its images fail at start), **0.1.115** (the control plane ran out of memory on a long-harvested estate; fixed in 0.1.116) and **0.1.127** (a credential named `client_token` could appear in clear; fixed in 0.1.128). 0.1.107 was superseded the same day by 0.1.108. A version whose build fails is void and is never retagged.

| Release crossed | Operator follow-up | Evidence to retain |
|---|---|---|
| 0.1.109 to 0.1.143 | An install older than 0.1.144 crosses these first: their rows are in [Post-upgrade checks, 0.1.109 to 0.1.143](/help/ops-upgrade-checks-0-1-109-to-0-1-143.md), oldest first, then continue here. | as that page lists per release |
| 0.1.144 | Schema, additive and created at start: new table `compliancereport` (one row per workspace per UTC day) and the JSON column `investigation.triage`. Re-run the AWS CloudShell setup for every account it set up (a role from the `deploy/aws` module: re-apply it). The read role gains `config:DescribeConformancePacks`, `config:DescribeConformancePackCompliance`, `ssm:DescribeInstanceInformation`, `ssm:DescribeInstancePatchStates`, `backup:ListBackupJobs`, `ec2:DescribeVpcEndpoints`, `ec2:DescribeTransitGateways`, `ec2:DescribeTransitGatewayAttachments`, `ec2:DescribeTransitGatewayVpcAttachments`, `ec2:DescribeTransitGatewayRouteTables` and `ec2:SearchTransitGatewayRoutes` (a role on `ReadOnlyAccess` already has them). The write role gains `ec2:CreateImage`, `ec2:CreateSnapshot` and `ec2:CreateReplaceRootVolumeTask` on instances and volumes tagged `aiops-managed=true`, the same three plus `ec2:CreateTags` on images, snapshots and restore tasks created with the `aiops:change` tag, and `ec2:DescribeImages`, `ec2:DescribeSnapshots` and `ec2:DescribeReplaceRootVolumeTasks`; nothing that deletes. Re-run the Azure Cloud Shell setup for a write connection: its role gains `Microsoft.Compute/disks/read`, `Microsoft.Compute/snapshots/read` and `Microsoft.Compute/snapshots/write`. The guided read role also gains `health:DescribeEvents` for the automated triage's cloud-health step (a role on `ReadOnlyAccess` already has it); until the setup is re-run, that step reads *not available*. Until the setups are re-run, **Scores** lists the new reads as denied and a reboot stops at its backup step (`UnauthorizedOperation`) before anything changes. Defaults that change behaviour on upgrade: a backup before every reboot-type step (`AIOPS_BACKUP_BEFORE_CHANGE`, on; kept 7 days by tag, never deleted by the platform; a reboot-type plan over a CLI write connection is refused at review); a certified step runbook carries its own operations (`AIOPS_RUNBOOK_OWNS_OPERATIONS`, on): **Pre-approved operations** covers single-call routes only, a fetched skill alone no longer makes a change standard, and pre-approving a fix waits until its incident is resolved; a person may only raise a change class, and a P1 a certified runbook covers is released under the standard rules (requests recorded before keep their stamped rules); denied access attempts open SecOps incidents while the SecOps agent is enabled (`AIOPS_DENIED_ACCESS_TO_SECOPS`, on); a scanner HIGH maps to `error` (P2) instead of `critical` (P1) (register rows change on the scanner's next send, open incidents keep their priority); a P3/P4 finding with a prepared fix waits for **Analyse**; the automated triage reads cloud health and known issues before each incident analysis and adds `## Fault domain` to its report (`AIOPS_TRIAGE_CHECKLIST`, on); the compliance reports are read once a day (`AIOPS_COMPLIANCE_REPORT_INTERVAL_S` = 86400). Off until a workspace turns them on: unauthorized-change incidents (`AIOPS_UNAUTHORIZED_CHANGE_INCIDENTS`; fill `change_authorized_principals` first), several runbooks per change (`AIOPS_RUNBOOK_MULTI_COVER`), the change management agent (`AIOPS_CHANGE_AGENT_CR`) and its fix drafting (`AIOPS_CHANGE_AGENT_OWNS_PLAN`); `AIOPS_PATCH_WITHIN_DAYS` is 0 (no rule). A private-network agent in database mode needs this release for the triage steps on its runs (in API mode the control plane does them); turning on the change management agent's fix drafting needs every private-network agent on this release. | The re-run setup output; the first **Scores** report with no denied call; `GET /guardrails` `switch_state` per workspace; one incident's **Automated triage** card |
| 0.1.145 | No schema change. Re-run the AWS CloudShell setup for every write connection it set up: the write role gains `iam:SimulatePrincipalPolicy` on its own ARN only, with an explicit deny on every other principal (a role from the `deploy/aws` module is unchanged and its permission check keeps reading *unknown*). Until the check can answer, the step-runbook automatic lane does not run a runbook on its own; a person releases it. Behaviour that changes on upgrade: an unconfirmed permission check stops an automatic step-runbook run; a change request the platform executed stops covering outside writes 60 seconds after its execution; STS calls are no longer harvested as changes; a fix whose alerts cleared at the source and whose validation plan reads as expected clears with no agent run; the agreed comparison table's sha256 changes (A.8.8 and A.8.13 notes), so a recorded acceptance reads as changed since and needs a new acceptance; a reviewed revert that re-creates a security-group rule with a description needs a new review (its digest now carries the description). | The re-run setup output; a step runbook's **Check runbook** showing the permission check passed; `GET /guardrails` `switch_state` per workspace |
| 0.1.146 | No schema change. Re-run the AWS CloudShell setup for every account it set up (a role from the `deploy/aws` module: re-apply it). The read role gains `lambda:ListFunctions`, `ecr:DescribeRepositories`, `ecr:DescribeImages` and `ecr:ListTagsForResource` (a role on `ReadOnlyAccess` already has them); until the setup is re-run, the harvest's **Coverage** lists `lambda:ListFunctions` as denied. The write role gains `ssm:SendCommand` with the `AWS-RunPatchBaseline` document only on instances tagged `aiops-managed=true`, `ssm:GetCommandInvocation`, `ssm:ListCommandInvocations` and `ssm:DescribeInstanceInformation`, and Reachability Analyzer (`ec2:CreateNetworkInsightsPath` and `ec2:StartNetworkInsightsAnalysis` only with the `aiops:change` tag on create, `ec2:StartNetworkInsightsAnalysis` on a path and `ec2:DeleteNetworkInsightsPath` and `ec2:DeleteNetworkInsightsAnalysis` only on what carries it, account-wide `ec2:Describe*` (instance and launch-template user data included), `ec2:GetManagedPrefixListEntries`, `ec2:GetTransitGatewayRouteTablePropagations`, `ec2:SearchTransitGatewayRoutes`, `elasticloadbalancing:Describe*`, `network-firewall:Describe*` / `List*` and `tiros:CreateQuery`, `ExtendQuery`, `GetQueryAnswer`, `GetQueryExplanation`, `GetQueryExtensionAccounts`); nothing uses those yet. `ssm:GetCommandInvocation` and `ListCommandInvocations` are on `*` and return any command's output in the region, and `AWS-RunPatchBaseline` takes a `BaselineOverride` and an `InstallOverrideList` IAM cannot restrict, so the write role can install packages on a tagged instance. The backup's `ec2:CreateTags` is now bound to the backup's own create calls (`ec2:CreateAction` = `CreateImage`, `CreateSnapshot`, `CreateReplaceRootVolumeTask`) and to the tag keys `aiops:change` and `aiops:retain-until`. A `deploy/aws` write role also gains `iam:SimulatePrincipalPolicy` on its own ARN and a region pin on its `aiops:change`-scoped statements. Defaults that change behaviour on upgrade: the scheduled Security Hub / Defender pull is on every 15 minutes (`AIOPS_SCANNER_SYNC_INTERVAL_S` = 900; a workspace that stored `0` stays off), whatever the SOC 2 / ISO 27001 switch says, and its Inspector CVE read is on unless a connection says `pull_vulnerabilities: false`. With **Analyse P1/P2 findings automatically** on, every HIGH or CRITICAL CVE starts one paid analysis (measured USD 1.04 to 2.92 each): before the first pull, set each workspace's `daily_cost_limit_usd`, set **Scanner sync** to off where the pull is not wanted, and give an AWS read connection into a shared account **Only findings on resources tagged** (`finding_tag_scope`, one `key=value`; any other value fails that connection with `invalid_tag_scope`, it never pulls wider). The change management agent drafts every change request (`AIOPS_CHANGE_AGENT_CR`, on): one change-agent run of at most 8 steps per change request, about USD 0.33 measured. A workspace that set `aiops_change_agent_owns_plan: true` without setting `aiops_change_agent_cr` now gets Phase 2 (the agent drafts the plan); set `aiops_change_agent_cr` or `aiops_change_agent_owns_plan` explicitly before the upgrade if that is not wanted. Screens and names: the change page's "Verification" card is **Validation plan** and "Safety net" is **Rollback plan**; "Alert correlation" is **Service correlation** in the sidebar and on L2 (skill id `alert-correlation`, the `/alerts` route and the `unmapped` filter are unchanged); the first firewall harvest lists deny rules as added once. API and MCP: the automated triage's `triage_checklist[*].step` numbers follow the new order (1 dependency map, 2 known issues, 3 cloud health, 4 recent changes, 5 signals, 6 summary, 7 fault domain); match a step by its `title`, not its number. | The re-run setup output; the next harvest's **Coverage** with no denied `lambda:ListFunctions`; a step runbook's **Check runbook** permission check; `GET /findings/sync` (`interval_s`, `interval_inherited`) and `GET /guardrails` `switch_state` per workspace |
| 0.1.147 | Schema, additive and created at start: new table `syntheticcheck`, the columns `incident.merged_into_id` and `investigation.scope_deferred` (JSON), and the index `ix_incident_tenant_opened` on `incident (tenant_id, opened_at)`. No new permission. New routes: `POST /api/aiops/{tenant_id}/changes/{change_id}/reject` (`approvals.decide`), `GET` and `POST /api/aiops/{tenant_id}/monitors`, `PATCH` and `DELETE …/monitors/{check_id}`, `POST …/monitors/{check_id}/run` and `POST …/monitors/derive` (`datasources.view` / `datasources.manage`), with the MCP tools `reject_change`, `list_monitors`, `create_monitor`, `update_monitor`, `delete_monitor`, `run_monitor` and `derive_monitors`. The storm gate is removed: `aiops_storm_threshold` and `aiops_storm_window_s` (a workspace setting, or `AIOPS_STORM_THRESHOLD` / `AIOPS_STORM_WINDOW_S` on the control plane) are accepted and ignored, and a burst of unrelated services now opens one incident each (the shared-alert-name fold, `aiops_fold_shared_cause_threshold` 3, stays). Grouping reads the new windows `aiops_storm_group_window_s` (300 s, at most 3600) and `aiops_storm_spread_window_s` (600 s, at most 7200), whole seconds, platform `AIOPS_STORM_GROUP_WINDOW_S` / `AIOPS_STORM_SPREAD_WINDOW_S` or a workspace setting; `aiops_storm_agent` keeps its meaning (the grouping agent, on). A `__storm__` incident left open is not split any more: no alert joins it, it resolves when its alerts clear or a person resolves it, and an alert on one of its services opens that service's own incident beside it. A dashboard or alert on `aiops_webhooks_total{outcome="storm"}` must read `outcome="grouped"`. SOC 2 / ISO 27001 are off in every workspace (`AIOPS_COMPLIANCE_FRAMEWORKS`, off): the first scheduler tick after the upgrade resolves every open platform-check finding (`control:<id>`, `by: scope-change`; one a person holds stays), and a compliance scan answers `409`. `AIOPS_COMPLIANCE_FRAMEWORKS` is platform-wide: `true` (helm `extraEnv`) turns SOC 2 / ISO 27001, EST-01 to EST-06 and the platform alarms back on in every workspace whose `compliance.enabled` is not an explicit `false`, customer workspaces included. Set it only once the customer has agreed the comparison table, never for a Test Lab twin on a shared control plane; Test Lab cases that expect the platform's own checks get no scan while it is off. Defaults that change behaviour on upgrade: the known-issue fast path (`AIOPS_KNOWN_ISSUE_FAST_PATH`, on: an alert-started analysis rated 100 % relevant runs capped at 3 steps); post-change validation in 3 rounds 300 s apart (a rule's `minutes` no longer times anything: `validation_rounds: 1` with `validation_interval_s` = minutes × 60 reads about the old single window); validation signals derived from the change request's plan when no rule matches (`AIOPS_VALIDATION_DERIVED`, on); a 10-minute hold on the automatic verify after a start, reboot or restart; daily synthetic-check suggestions (`AIOPS_MONITOR_DERIVE`, on): up to 20 internet-facing load balancers and API Gateway APIs per workspace are probed from the control plane, `GET /` every 300 s, in trial (nothing opens) until a person keeps or discards them; an API Gateway call is billed and can invoke its integration, so turn `aiops_monitor_derive` off where that is unwanted; backup findings from the daily AWS Backup read (`AIOPS_BACKUP_FINDINGS`, on: one P3 incident per workspace while SOC 2 / ISO 27001 are off); CI/CD service-account writes recorded as changes (`AIOPS_K8S_CI_NAMESPACES`: `flux-system`, `jenkins`, `gitlab-runner`, `tekton`); Azure DevOps `workitem.updated` deliveries that change the state are signals, so a service hook that already sends them now resolves incidents on Done, Closed, Resolved or Removed, and a state change on a work item the platform has not seen opens an incident with an analysis (any other update is ignored; an incoming ticket missing required fields is not held yet); AWS Config findings key on their own rule (`control:<rule-name>`): a finding already open in the shared incident stays there until it clears, new ones open one incident per rule. Roll private-network agents to this release: an agent in database mode needs it for the service's resources in the prompt, a run's resource selection and the vulnerable-package plan per resource type. | `GET /guardrails` per workspace: `switch_state.known_issue_fast_path`, `verification.validation_derived` and `compliance_frameworks_available: false`; the platform-check findings resolved `by: scope-change`; the first `incident.grouped` audit row; **Synthetic checks** after the first daily suggestion; agent version |
| 0.1.148 | No schema change or permission. New workspace setting `aiops_grouping_wait_s` (whole seconds `0`–`600`, default 300, `0` = no grouping wait; `PATCH /api/aiops/{tenant_id}`), separate from `aiops_storm_group_window_s`: a workspace that raised the group window keeps a 300 s wait. New audit action `change.approval_lapsed`. New route `POST /api/aiops/{tenant_id}/changes/{change_id}/close-without-verdict` (`approvals.decide`, body `{"reason": "…"}`; `409` until the change has gone 24 hours without a verdict), with the MCP tool `close_change_without_verdict`. An approved change released by a person who did not vote runs again; a change the 0.1.147 dispatch re-check set to `rejected` stays rejected, so its incident needs a new change request. Behaviour that changes on upgrade: an incident on a service that depends, over a dependency allowed to fold, on another service holds its analysis for the grouping wait (`aiops_grouping_wait_s`, 300 s) while its ticket and notification go out as usual, and an alert that clears inside the wait resolves its incident with no analysis (when it comes back it is analysed at once); with a reopen window set, an incident a person closed as known noise is no longer reopened (the next alert on its key opens a new incident, analysed as usual), and a caller whose resolved incident would reopen joins its root's open incident instead. One-shot clean-up of the connection tag scope (**Only findings on resources tagged**): AWS Backup records already open for a resource outside it clear at the next daily read of their region (one `compliance.backup_out_of_scope` audit row; a tag none of the resources read carries clears nothing and the **Backup** card says so); suggested synthetic checks nobody kept whose resource is outside it are removed at the next daily suggestion run, also where suggestions are turned off (named in the `monitor.derived` row); built-in detection pairs already in alarm on such a resource resolve (*outside the workspace scope or the connection's tag scope*), except under a scope that cannot be read or that none of the connection's resources carries, which resolves nothing. Security Hub findings of DynamoDB.1, the deletion-protection controls, ElastiCache.1/.2, FSx.1/.2 and Redshift.3 move family when Security Hub next sends them (daily). The rollback sweep no longer escalates a change that stopped at its backup. The fixes run in the control plane; roll private-network agents as usual. | An approved change released by a second approver reaching `change.executed`; a dispatch refusal's `change.release_refused` row naming its reason; the first `compliance.backup_out_of_scope` row and the **Backup** card's count; a caller incident reading **Held for triage — grouping window: …**; agent version |
| 0.1.149 | Image and chart bump, no values change; no new table or column. The start's schema step no longer scans tables: each statement waits at most 3 s for a table lock and retries (5 tries, the last waiting up to 60 s); with leader election off the start still resumes the runs the restart interrupted before it serves. Once the app serves, the control plane builds seven new indexes with `CREATE INDEX CONCURRENTLY` (`ix_auditevent_action_target_at`, `ix_auditevent_tenant_target_at`, `ix_incident_tenant_key_status`, `ix_incident_tenant_status_resolved`, `ix_investigation_tenant_started`, `ix_knowledgeitem_tenant_name_version`, `ix_changerecord_live_status`; minutes on a large audit trail, writes continue) and runs `migration:null-defaults-v1` once ([Schema changes after the pod is ready](/help/ops-upgrade-and-verify.md#schema-changes-after-the-pod-is-ready)); both need a direct database connection, not a pooler in transaction mode. A workspace without a PII salt gets one at the first start. The chart adds a startup probe (60 × 5 s) before liveness counts. A webhook body over 1 MiB is answered `413` (Alertmanager, Grafana and the generic intake: 8 MiB; `kubernetes-audit`: 16 MiB) and the sender does not retry it: where a source posts larger groups, raise `AIOPS_WEBHOOK_MAX_BODY_BYTES` through `extraEnv` or set Alertmanager's `max_alerts` ([Webhook body cap](/help/ops-upgrade-and-verify.md#webhook-body-cap)). `aiops_change_records` exports open statuses only: a dashboard or alert on `completed`, `rolled_back`, `rejected` or `recorded` reads nothing. New series: `aiops_scheduler_subtick_seconds`, `aiops_migration_pending`. Rollback: the previous image with this chart kept ([Rollback](/help/ops-roll-back-a-release.md)); nothing in the database is undone and the new indexes stay. | The `migrations` block of `GET /api/aiops/admin/overview` with `pending: 0` (every index `valid`, `null-defaults-v1` `done`); `/healthz` version; the pod's time from start to Ready (under 30 s when no run was interrupted); every `413` line in the control-plane log, with its source |
| 0.1.150 | No schema change, setting, route or permission. Behaviour that changes on upgrade: a step runbook's or the analysis's fix with a step on a resource that only an alert merged, folded or grouped into the incident names (and none on the incident's own) is never a standard change, and its release is refused, by a person (`403`, `change.release_refused` or **Fix failed** reading *refused: …*) or automatically (`route.match_skipped`, reason `target_not_own`, also for a direct API route whose fixed target is such a resource); a change request drafted before the upgrade keeps its class, but its release is refused the same way: re-analyse that incident. A clear of an incident whose executed change acted only on such a resource completes the change with the verdict `uncertain` and no last-known-good, memory or runbook credit, and writes `incident.fix_not_credited`; a change credited before the upgrade keeps its verdict. A validation round is clean only when every one of the incident's own alarms passed: an own alarm that read no data fails the round when it treats missing data as breaching and CloudWatch reads it in ALARM (the read role's existing `cloudwatch:DescribeAlarms`), and otherwise leaves the round not observed (an alarm that treats missing data as not breaching reads OK); a window opened before the upgrade, whose alarms all count as the incident's own, reads a round with a silent alarm as not observed instead of clean, and such rounds hand the fix to the agent verify. An incident alarm on a count that publishes nothing while healthy (errors, 5xx) and keeps CloudWatch's default `TreatMissingData` (`missing`) now leaves its rounds not observed: set it to `notBreaching`. An approval and the run of a change are now two ledger entries (`approved`, then `released` by the person who ran it); entries written before the upgrade are unchanged. A reopen chooses the suspected change again (`incident.suspect_repinned`, trigger `reopen`). Roll private-network agents in database mode: they draft the change request, its class and scope. | On an incident holding a merged-in alert, a fix on the merged-in resource: its class reason and its review naming that resource, and no release offered; the next validation round listing every V-item; the change request document of a two-person release listing *Approved by* and *Released by*; agent version |
| 0.1.151 | No schema change, setting, route or permission. Behaviour that changes on upgrade: an incident's fix estimates its blast radius with the incident's service in every review (with a harvested estate, a step the estimate cannot model exactly can also name the one resource carrying that name), so an approval recorded on the incident page before the upgrade for such a fix no longer matches its digest and is asked for again; nothing ran under it. Once a step of a released change has run, the check before each later step keeps the blast radius the release was approved on. A certified step runbook's automatic release that ran no step, or stopped before a later one, writes `incident.auto_remediate` with `not_dispatched` or `stopped_before` and `reason`; rows written before the upgrade keep reading *a step failed*. Every incident's error start, ranking resource, alarm read, RCA Timeline start and triage resources now come from the alert that opened its current episode among its own alerts, and the platform's own fix for the incident is never ranked, so the **Probable cause** card of an incident that absorbed another service's incident can rank differently on its next read. A reopen stamps `previous_end`, `previous_from` and `resolution_sig` on `incident.recurrence`: an incident reopened after the upgrade is ranked from where its previous episode ended and drops the earlier episode's resolution note at its next close; one reopened before the upgrade has no stamp: its window reaches back to the previous error on its key or the last verified fix, else the look-back cap, and it keeps that note. A validation window still running at the upgrade also gets the added round, and the scheduled check takes a `remediating` incident before the other incidents of its workspace. Model calls go over httpx instead of litellm's aiohttp transport: `HTTPS_PROXY` and `NO_PROXY` still apply, and `NO_PROXY` entries are matched by httpx's rules. The fixes run in the control plane; roll private-network agents as usual: an agent's model calls move to httpx once it runs this release, and an agent in database mode needs it for the triage's own-alert resources on its runs. | The next automatic step-runbook release reaching `change.executed`, or its Timeline row reading **Automatic fix refused before dispatch** with the reason; `cause.window.floor_basis` `previous_episode` with `floor_incident_id` the incident itself in `GET /api/aiops/{tenant_id}/incidents/{id}/lkg` of an incident reopened after the upgrade; no `Unclosed client session` line in the control-plane log; agent version |
| 0.1.152 | No schema change, setting, route or permission. Behaviour that changes on upgrade: a change harvested since a verified fix on the resource it touched now counts against the known-issue relevancy whether the ledger names the resource by its bare EC2 id or its ARN, and a harvested change counts for an incident by its resource or its linked services, never by its event source: an incident whose fixed instance or volume changed since the fix reads 85 % and its analysis runs in full where it took the fast path before. The same match widens the record's *changes on these resources since that resolve* on the incident page and in the analysis's PRIOR RESOLUTION. The relevancy recorded on a triage gains `fix` (`held`, `did_not_hold`) and the fast-path instruction says whether the prior fix held; a run started before the upgrade and restarted after it says only *a verified fix*. Triage step 2, the header chip and the incident list read `recurring ×N` / `recurring` on a new incident of a resolved alert; `recurrence.kind` stays `reopened`, and the incident detail's `recurrence` gains `kind_at` (an in-place reopen). The **Probable cause** card shows the window's own look-back sentence. The fixes run in the control plane; roll private-network agents as usual: an agent in database mode computes the relevancy on its runs. | On the next new incident of a resolved alert, the header chip reading `recurring ×N`; `investigation.triage.known_issues.relevancy.fix` in `GET /api/aiops/{tenant_id}/investigations/{id}` of its triage run; agent version |
| 0.1.153 | Schema, additive and created at start: the JSON columns `detectionstatus.rule_since` and `detectionstatus.baselines`. No new route. **Before upgrading, ticket intake:** a Jira, ServiceNow or Azure DevOps ticket sent in must now name service, severity, priority, assignee and notify (`AIOPS_TICKET_REQUIRED_FIELDS`, a JSON list, default all five; workspace `ticket_required_fields`, `[]` = none of the five — a ticket naming no service still waits while `aiops_unmapped_triage` is on, the default); one missing any waits in the triage queue (no analysis, notification or automatic fix) and AIOps asks on it once through the workspace's **Ticket system** route. Update the ServiceNow inbound business rule to the new template (it posts severity, impact, urgency, assignee, assignment group and watch list, and fires on **Update** as well as Insert), name the severity and who to notify on Jira issues (Jira has no field for either: `Severity:` / `Notify:` line or `severity:` / `notify:` label), tag Azure DevOps work items `service:<name>` and `notify:<team>` (or the same description lines), and enable Jira `jira:issue_updated` and Azure DevOps `workitem.updated` events so an update can move a held ticket on — or narrow `ticket_required_fields` for each workspace first. Re-run the AWS CloudShell setup for every account it set up (a role from the `deploy/aws` module: re-apply it): the read role gains `rds:DescribePendingMaintenanceActions` (built-in detection R36; a role on `ReadOnlyAccess` already has it), and the write role gains `ec2:ModifyNetworkInterfaceAttribute` on network interfaces tagged `aiops-managed=true` and on any security group in the region, for the EC2 patch runbook's isolation and restore (the group side cannot be narrowed by tag, so the role can attach any group in the region, an open one included, and the same action changes other interface attributes such as `SourceDestCheck`), and `ssm:DescribeInstancePatchStates`. Tag the instances' network interfaces `aiops-managed=true` too. Until the setup is re-run, an isolation stops at its first interface with `UnauthorizedOperation`, nothing changed. New setting `aiops_patch_findings` (`AIOPS_PATCH_FINDINGS`, on; a workspace override): the daily Patch Manager read lands each non-compliant node in the tag scope as a **Patching** finding (control `PATCH.COMPLIANCE`, one incident per workspace; HIGH when a critical patch is missing, which is P2, else MEDIUM, P3), unless Security Hub's SSM.2 already holds that node open. With **Analyse P1/P2 findings automatically** on, the first read after the upgrade can start one paid analysis per workspace, and the incident's ticket and notification go out on open; set `aiops_patch_findings: false` before the upgrade where that is not wanted. The demo workspace gains the disabled example route *EXAMPLE: EC2 patch with backup and isolation* on a disabled placeholder write connection *example-aws-writer*; other workspaces create the route from the request in the runbook chapter, named `remediate-…`. New runbook operations, used only by a runbook that carries them: `aws.ssm.patch_baseline`, `aws.ec2.isolate` and `aws.ec2.restore_sgs`. Built-in detection gains R26–R37 (latency and fault baselines, EKS, Kubernetes pod and node state through a read-only kubectl connection, EC2 and RDS scheduled maintenance at P4 (severity `info`), AWS Health issues the estate is exposed to at P2): on a workspace already armed each new rule watches **in shadow** for the shadow period (24 h by default) from the first poll it covers something, then arms itself; **Arm now** arms them at the next poll. A built-in `error` alert (R37) is analysed, ticketed and notified like a critical one. New audit actions `ticket.info_requested`, `ticket.info_received`, `change.isolation_left` and `compliance.patch_out_of_scope`; new metrics `aiops_detection_episodes_total` and `aiops_detection_reads_total`, and `op="ask"` on the ticket metric. The changes run in the control plane; roll private-network agents as usual. | The re-run setup output; a test ticket with a field missing waiting in the triage queue with its question on the ticket; the detection card's per-rule shadow line; the next daily **Patching** card and its findings; agent version |
| 0.1.154 | No schema change, no new route. Re-run the AWS CloudShell setup for every write connection it set up (a role from the `deploy/aws` module: re-apply it): the write role gains `codebuild:StartBuild` and `codebuild:BatchGetBuilds` on the one CodeBuild project `aiops-image-rebuild` in the connection's region while it carries `aiops-managed=true`, `ecr:DescribeImages`, and explicit denies of the `StartBuild` overrides that change what the build runs (buildspec, source, secondary sources, build image, compute, environment type, privileged mode, fleet, and any variable but the five the platform sends), for the container image rebuild. The project is the customer's: create it with `deploy/aws` `image_rebuild = true`, tag each repository it may push to `aiops-managed=true` and make those repositories' tags immutable. Until the setup is re-run, a rebuild stops at its start with `AccessDeniedException` before anything is built. The opt-in pull block (`aiops_ecr_block_pulls`, **Allow a runbook to block pulls of a vulnerable image**, off) also needs `ecr:GetRepositoryPolicy`, `ecr:SetRepositoryPolicy` and `ecr:DeleteRepositoryPolicy`, which only `deploy/aws` with `image_pull_block = true` grants. With `image_pull_block = true`, more than six regions (fewer with a tag key longer than `aiops-managed`) put the `deploy/aws` write policy over IAM's 10,240-character limit and the apply fails. A write role set up before 0.1.146 also lacks the Reachability Analyzer statements the neighbour reachability check uses (`aiops_reachability_checks`, API only, off; at most `aiops_reachability_max_per_change` analyses a change, 10, about USD 0.10 each billed to the AWS account). Both switches stay off until a workspace turns them on. New setting `aiops_image_reminder_h` (`AIOPS_IMAGE_REMINDER_H`, 24, `0`–`720`), new notification event `image_not_deployed` (a target that lists its events gets it only once ticked; it repeats until the scanner stops reporting the old digest — moving the workloads alone does not stop it: delete or untag the old digest once nothing runs it), new audit action `image.not_deployed`, new runbook operations `aws.ecr.rebuild_image` and `aws.ecr.block_image_pulls`. Behaviour that changes on upgrade: the step-runbook permission check reads an action the write role lacks as `missing` — IAM's list of condition keys on a deny, implicit or explicit, no longer makes it `unknown` — so certification is refused where it went through before; a route certified before the upgrade keeps its certification, but its next **Check the write connection's permissions** and its re-certification read `missing`, and so does the check before an automatic run, except on the instance's network interfaces, whose tags it does not read (`unknown`: a person releases the change). A `deploy/aws` role scoped by a tag other than `aiops-managed=true` reads `missing` at certification too, until the write connection's Scope JSON names that tag: paste `scope` from the re-applied module's `datasource_snippet`, which now carries `resource_tag`. The changes run in the control plane; roll private-network agents as usual. | The re-run setup output; **Check the write connection's permissions** on each certified step runbook reading *every permission is granted*; agent version |
| 0.1.155 | Schema, additive and created at start: the partial index `ix_changerecord_live_status_v2` (`not_needed` is a new terminal change state; the 0.1.149 index stays). No new setting or route. Re-run the AWS CloudShell setup for every write connection it set up (a role from the `deploy/aws` module: re-apply it). The write role's statements no longer fit IAM's 10,240-character limit for a role's inline policies, so the backup and restore, Reachability, image rebuild and Lambda runtime statements move into four customer-managed policies attached to the write role only; the inline policy keeps the rest and the Deny boundary, and no statement's scope changes. The re-run's change set adds the four policies and modifies the write role: the stack update creates them, there is nothing to delete first, and the setup's review accepts only the template's own roles and policies, added or modified, never replaced or removed. Whoever runs the re-run (or applies `deploy/aws`) now also needs `iam:CreatePolicy`, `iam:GetPolicy` and `iam:AttachRolePolicy`, and `iam:CreatePolicyVersion` and `iam:DeletePolicyVersion` once a later release changes a policy (Terraform also reads `iam:GetPolicyVersion` and `iam:ListPolicyVersions`); an installer scoped to roles only sees the stack update fail and roll back, the role unchanged. `deploy/aws` creates `<name_prefix>-write-backup`, `-reachability`, `-image-rebuild` and `-lambda-runtime` and attaches them (four of the ten managed policies IAM attaches to a role by default); its write policies now fit every commercial region, with or without `image_pull_block`. The write role gains: the restore of an image backup — `ec2:CreateReplaceRootVolumeTask` from this account's own image or snapshot carrying `aiops:change`, into a new root volume only, its task tagged on create (0.1.144–0.1.154 allowed the image only with a request tag an existing image never carries, so AWS refused **Roll back** of the EC2 patch runbook; until the setup is re-run it still does, with `UnauthorizedOperation` before anything changes, and **Check the write connection's permissions**, the check before an automatic run and re-certification of the patch runbook read `ec2:CreateReplaceRootVolumeTask` missing); `ec2:CreateNetworkInsightsPath` on the network interfaces in the region (until re-run every neighbour reachability row reads *not observed* with `UnauthorizedOperation`); and `lambda:PublishVersion`, `UpdateFunctionConfiguration`, `UpdateAlias`, `GetFunctionConfiguration` and `GetAlias` on functions tagged `aiops-managed=true` (the Lambda runtime upgrade; until re-run it stops at its backup with `AccessDeniedException`, nothing changed). A CodeBuild project `aiops-image-rebuild` created by `deploy/aws` before 0.1.155 runs the old buildspec, which refuses a source package such as `expat`, until `deploy/aws` is re-applied. Switches and defaults: no setting or default changes. With both automatic-fix switches on, a compliance control, scanner finding, vulnerability or security alert now waits for a person on every lane, a certified step runbook's included. `aiops_reachability_checks` (off) also adds the reachability permissions to a runbook's permission check when on. The demo workspace gains the disabled example route *EXAMPLE: Lambda runtime upgrade with version backup*. A Jira or Azure DevOps ticket route fills the component, assignee or priority only after **Check** is run on it and the route saved again. **Send test ticket** works on a disabled route (it answered 409) and never enables it. Behaviour that changes on upgrade: a step runbook's change request is drafted at the tier its release needs (L2 for its API steps, L3 with a pull block) instead of L1; the change request document prints the runbook's own rollback steps; the patch runbook's post-change check reads the patch command's status and Patch Manager's missing and failed counts, and the restored groups against those the isolation recorded, instead of the instance's power state (a read that cannot answer is *not read*, so the paid verify turn runs), and a change drafted before the upgrade that still lists the isolation phase reads it by the restored groups; a change whose only target already complies ends `not_needed` (audit `change.not_needed`, metric outcome `not_needed`; rows that landed `failed` before stay failed); the change-window rule no longer folds a scanner finding into another incident, or an alert into a finding's incident; tickets that join a live incident are counted as `joined`, not `suppressed` (webhook response and `ingest.batch` audit); a kubectl cluster's estate ids and built-in detection alerts carry the connection's name — the next harvest removes the nodes written as `k8s:cluster/…`, and an alarm under such an id keeps clearing; EKS service-linked-role writes and `kubectl auth whoami` are no longer recorded (counted as `service`) and a pipeline's refused write is kept as a denied attempt; **Test** on a private FortiGate or PAN-OS management address answers `inside_network`; a FortiGate connection gains the read-only `<name>_traffic_log` tool. The changes run in the control plane; roll private-network agents as usual: an older agent does not offer certified step runbooks to a finding's analysis. | The re-run setup output listing the four managed policies (Add) and the write role (Modify), or the `deploy/aws` plan's four `aws_iam_policy.write` and their attachments; **Check the write connection's permissions** on the certified patch runbook reading *every permission is granted*; agent version |
| 0.1.156 | Schema, additive and created at start: the column `runnertoken.serves_chat`; at that start every agent token that had already connected (the provisioner's aside) is marked as serving chat, so no private chat moves. New routes: `POST /api/aiops/{tenant_id}/itsm-webhook-token`, `POST /api/aiops/{tenant_id}/k8s-audit-token/{cluster}`, `POST /users/{user_id}/reset-link`, `POST /api/aiops/{tenant_id}/changes/stats` and `POST /api/aiops/{tenant_id}/benchmarks/model-report`. New settings: `AIOPS_METRICS_TOKEN` (control plane; empty = no check, the chart sets it) and `AIOPS_KUBECTL_IN_CLUSTER` (off; set only by the private agent chart's `inClusterRbac` mode). No switch default changes. **Before the chart upgrade, metrics:** the metrics listener answers `401` without `Authorization: Bearer <AIOPS_METRICS_TOKEN>` as soon as the control plane restarts, so first give every scrape job of port 9464 an `authorization` block reading the `AIOPS_METRICS_TOKEN` key of the release Secret (an annotation-driven job too: annotations carry no token), then upgrade; with `existingSecret` the listener stays open until that Secret holds `AIOPS_METRICS_TOKEN`: make Prometheus send the token first, then add the key, then set `metrics.serviceMonitor.bearerToken: true` if you use the ServiceMonitor. **After the upgrade, change integrations (two breaking changes):** (1) ServiceNow and Jira change tickets: for every workspace with `aiops_unauthorized_change_incidents` on, mint the ITSM webhook token — `POST /api/aiops/{tenant_id}/itsm-webhook-token` with `members.manage` (an `admin`-scope personal access token, or MCP `mint_itsm_webhook_token`; the token is shown once, minting again invalidates the previous one) — and re-point the ServiceNow business rule or Jira automation that forwards change tickets to send it as `Authorization: Bearer <token>` on `/webhook/servicenow/<workspace-slug>` or `/webhook/jira/<workspace-slug>` (alerts from the same webhook keep working with either token). Until then a ticket sent with the shared webhook token no longer accounts for a captured write, so the writes it covered are flagged as unauthorized; a ticket recorded earlier counts again on its next re-send with the token, and tickets sent through your own change feed never count. (2) Self-managed clusters' audit webhooks: for each cluster, mint its token — `POST /api/aiops/{tenant_id}/k8s-audit-token/<cluster-name>` with `pillars.manage` (an `admin`-scope personal access token, or MCP `mint_k8s_audit_token`; the portal has no control for it), `<cluster-name>` being the last path segment the cluster already posts to — put it in the audit webhook kubeconfig's `token:`, keep the `server:` URL ending in `/-/<cluster-name>` and restart the kube-apiserver. Until then the API server's deliveries with the webhook token are refused (`401`), the form without the cluster segment (`…/kubernetes-audit/<slug>` with or without `?cluster=`) answers `404`, and the writes they carry are not recorded; the API server drops what it cannot deliver. EKS clusters read through `k8s_audit: true` on the AWS connection need nothing ([Kubernetes audit webhook token](/help/ops-upgrade-and-verify.md#kubernetes-audit-webhook-token)). **Connections:** a kubectl connection whose kubeconfig runs a credential plugin (`exec` or `auth-provider`: an EKS, GKE or AKS default kubeconfig) or names a file on the platform host (`tokenFile`, `client-certificate`, `client-key`, `certificate-authority` as a path) answers `This kubeconfig runs a credential plugin … or names a file on the platform host …` from every tool call, **Test** and inventory read until it is saved again with a static or service-account `token` and embedded data (`certificate-authority-data`, `client-*-data`; `kubectl config view --raw --minify --flatten` embeds them); a kubeconfig that resolves to no server (`{}`, no current context) now fails instead of reading the platform's own cluster. A Google Cloud connection saved without a service-account key fails closed: add the key (`PATCH /api/aiops/{tenant_id}/datasources/{ds_id}` with `secret_values`) or disable it. A keyless AWS connection saved with a `${field}` placeholder in a key field fails closed until it is saved again, which binds the platform-minted External ID. Changing a write connection to another role, or the config of a write connection that holds a stored secret, now needs `approvals.decide` as well as `datasources.manage` (`PATCH /api/aiops/{tenant_id}/datasources/{ds_id}`): the builtin operator is refused `403`; an owner, approver or executive is not. The ssh connector's `--` lands on existing rows at the next start. **Accounts and tokens:** platform staff can no longer set a password on an account that holds a workspace membership or an organization seat, or that was born from single sign-on (`409`); use `POST /users/{user_id}/reset-link` (platform administrator, `admin`-scope token; needs a mail relay, the link goes to the account only) or let the workspace owner use **Reset password** on the member's row (`POST /api/aiops/{tenant_id}/invites/{user_id}/reissue`; emailed when a relay exists, else shown once). An account created by single sign-on that is not platform staff no longer signs in with a password: tell such users before the upgrade. A password change, reset or `AIOPS_ADMIN_FORCE_PASSWORD` now ends the account's other sessions. A personal access token now needs the `admin` scope for member seat and role changes, `PUT …/idp`, group-admin workspace staffing and the three token mints: re-mint a `write`-scope token a script uses for those. A private agent that should answer workspace chat is created by an owner with **Answers workspace chat** (`members.manage`); tokens that had connected before keep serving chat; staff-minted and operator-minted tokens never do. With `AIOPS_SEED_DEMO` off, the published demo webhook token is refused on every workspace: a workspace that still sends with it mints its own (`POST /api/aiops/{tenant_id}/webhook-token`). **API callers:** a script that saved a report with `GET …/changes/stats?export=true` or `GET …/benchmarks/model-report?export=true` now gets the read-only report; call `POST` at the same path with a `write`-scope token. A custom role that reads `GET …/compliance/discovery-findings` needs `datasources.view` too (every builtin role has it). `aiops_compliance_muted` in `PATCH /api/aiops/{tenant_id}` answers `400`. **Behaviour that changes on upgrade:** a write an audit log captured is judged against the freeze window at the provider's audit time and a pushed or ticketed change at its receipt, so a write made in a freeze and pulled after it is flagged where it was not before; an L3 in-run approval counts a vote only for the command it released and only while the voter may still release the tier, so a pending L3 release with a stale or mismatched vote asks again; `approval.*` audit payloads written from now on carry the invocation with credentials hidden (older rows are shown hidden, the export is unchanged); a platform change no longer pins or folds alerts on its impacted-services list; a post-change round derived from a CloudWatch alert reads the alarm at the source, and an alarm CloudWatch does not confirm only flags a round; a rollback the provider refuses before any write releases its reservation and **Roll back** is offered again (a change locked that way before the upgrade stays locked); the platform's own Lambda runtime write is recorded under the function ARN; a deactivated workspace's (and its lab box's) failed changes wait for Activate instead of being rolled back automatically; a response from a workspace-chosen endpoint past 2 MiB or past the deadline is an error (a ticket system's or a git host's included: the ticket step fails, a write that got it is uncertain), and a lakehouse pull past 16 MiB an error status. The changes run in the control plane and in the agents: roll private-network agents as usual — an older agent still runs kubeconfig credential plugins, reads uncompressed answers without a cap and sends a compiled step re-rendered; its approval receipts are filtered on the control plane whatever its version. | The ITSM webhook's next change ticket recorded with `itsm_verified` (`GET /api/aiops/{tenant_id}/changes/{id}`); a `200` on each cluster's next audit batch and its writes on the Changes page; `curl` of `/metrics` answering `401` without the token and `aiops_build_info` with it; a kubectl connection's **Test** passing after the re-save; agent version |
| 0.1.157 | Schema, additive and created at start: the column `cloudfinding.cleared_reason`, with a one-shot backfill (`migration:cleared-out-of-scope-v1`: a Backup or Patch Manager record the platform's read cleared in the 15 minutes before its out-of-scope audit row is marked `out_of_scope` and no longer counted as fixed). New routes: `GET /api/aiops/{tenant_id}/runbook-templates`, `GET /api/aiops/{tenant_id}/incidents/summary`, `GET /api/aiops/{tenant_id}/changes/summary`, `POST /api/aiops/{tenant_id}/changes/{change_id}/schedule` and `POST /api/aiops/{tenant_id}/changes/{change_id}/schedule/cancel`; no new permission. New workspace switches with their platform defaults: `aiops_demote_on_deviation` (on), `aiops_canary_execution` (off), `aiops_repeated_other_resource` (off); the settings map `patch_schedules`; the notification event `change_schedule`. No chart value changes. **Nothing to do for the upgrade itself.** Optional: re-run the guided AWS setup only for a shared account that wants the writer without the three account-wide statements (**Leave out account-wide settings (shared account)**; the choice is not remembered on a re-run, so pick it each time). **Behaviour that changes on upgrade:** a certified step runbook whose backup or change target is a literal resource, or whose selector names one exact resource, no longer expands or classes a change (the route list and Needs you show the finding; give it a parameter bound from a label or annotation and certify it again); with `aiops_demote_on_deviation` on, a run through a certified runbook route that fails its validation or whose fix does not hold takes the route's certification away until an approver certifies it again (`PATCH /api/aiops/{tenant_id}/guardrails` `demote_on_deviation: false` keeps the 0.1.156 behaviour, where only a rollback demotes); every executed write step of a multi-step plan is followed by a read-only read of the resources it wrote (at most 10 a step, 40 a change) through the step's own connection, each a `snapshot` artifact — more read calls on the write role; the analysis prompt grows by the verdict-criteria line of the change candidates block (2,500 to 2,900 characters) and, when certified step runbooks match the incident, by a block of up to three runbooks, at most 900 characters each; a compliance or vulnerability change can be scheduled, and the scheduler releases it as the person who scheduled it with their role at that time (a removed or re-roled person's schedule lands refused, open to schedule again); a package-upgrade step needs `ssm:SendCommand` on `AWS-RunShellScript` on the write role (the guided role does not grant it: add it before certifying such a runbook). The demo workspace gains five disabled, uncertified step runbooks captured from Automated Security Response on AWS. Private-network agents need no change: the prompt additions reach an api-mode agent through the run's context from the control plane, and execution stays on the control plane. | `GET /healthz` reporting 0.1.157; `GET /api/aiops/{tenant_id}/runbook-templates` answering seven templates; **Check runbook** reporting `pinned_target` on a runbook with a literal target; the Changes list's dashboard counts; the first `change.scheduled` audit row |
| 0.1.164 | No schema change, setting, route, permission or chart value change; new audit actions `lkg_record.outside_explained` and `change.class_restored`. **Nothing to do for the upgrade.** **Behaviour that changes on upgrade:** a runbook demoted for a recurrence whose outside write is harvested later is certified again (`route.deviation_explained`, `restored: true`) and that incident's unrequested change requests go back to `standard`; a demotion from before 0.1.164 waits for an approver. Private-network agents need no change. | `GET /healthz` reporting 0.1.164 |
| 0.1.163 | Schema, additive and created at start: the column `cloudfinding.failing_since` (no backfill: a finding already failing names no change until its next failing episode), and the one-shot `migration:finding-family-v1` (classifies every stored finding again, 500 rows a batch, and writes the family only where it moved: DynamoDB.1, DynamoDB.6, ELB.6, Cognito.6 and RDS.8 rows leave the **Backup** list). New setting `AIOPS_REQUIRE_BEFORE_STATE`, the platform default of `aiops_require_before_state` (on). New audit actions `incident.change_linked`, `change.finding_linked`, `compliance.finding_gone` and `compliance.scanner_out_of_scope`. API additions: `GET /api/aiops/{tenant_id}/alerts` takes `q` and `service` (at most 200 characters each, else `422`); `GET /findings` rows carry `failing_since`, the change detail `upgrade_warning`, and the harvest report and the manual pull's answer `findings_linked`. No new route, permission or chart value. **Two platform defaults turn on:** `aiops_require_before_state` and `aiops_hold_for_approval`, for every workspace that never set them. With the before-state rule on, a direct write runs only after its before-state read: a typed step over a CLI connection and a legacy direct route (`api`, `cli` or `mcp`) that declares no snapshot read become manual-only, and an approved or scheduled change on such a route is refused when it fires (*the reviewed action changed: Manual route only…*); pull requests and rollbacks are exempt. With the hold on, an escalated incident whose change request was requested (drafted, pending approval, or approved but not run), or whose proposal matched a certified runbook, no longer closes on the quiet window or when its ticket closes at the source; the first sweep audits `incident.autoclose_held` once per held incident. To keep the 0.1.162 behaviour, `PATCH /api/aiops/{tenant_id}/guardrails` `{"hold_for_approval": false, "require_before_state": false}`. A workspace that stored a value keeps it (on 0.1.162 an unticked before-state box stored `false` and read the same as unset): to follow the new default, send `{"hold_for_approval": null, "require_before_state": null}` and check that `switch_state` shows `override: null` for both. **Behaviour that changes on upgrade:** the next scanner pulls clear open Security Hub findings the hub no longer answers (last seen over 48 hours ago, every region read in full; a finding of another account than the connection's own only when the same pull returned a finding of that account, else once last seen over 90 days ago) and, while every pulling AWS connection has a tag scope, findings on harvested resources outside it; their incidents resolve, with the resolve notification and ticket write-back. A tag-scoped connection's Inspector read lands the CVEs on tagged repositories and functions, so a P1 or P2 one may start an analysis (`aiops_auto_investigate_p1p2`). R31 also fires between back-offs; a scoped workspace's denied attempts outside its scope open no SecOps report; a built-in metric alert joins its resource's `service` tag; a release whose quorum lapsed waits for approval again instead of being rejected; a `finding_tag_scope` without one `key=value` pair is refused at save (`400`). **Thresholds and why** is its own chapter, and old `settings-reference#thresholds-*` links land on its pointer section. The changes run in the control plane; a private-network agent on an image before 0.1.163 may leave out the change agent's upgrade warning, which then reads *unknown, not checked*. | `GET /healthz` reporting 0.1.163; the control-plane log line *online schema: finding-family-v1 refiled N finding(s)*; `switch_state` on `GET /guardrails` listing `require_before_state`; after an outside write and its scanner finding, the write's id in the incident's `change_refs` |
| 0.1.162 | No schema change, setting, route, permission, audit action or chart value change. **Nothing to do for the upgrade.** **Behaviour that changes on upgrade:** the findings list fits its card from 1280 px, the runbook catalog at 1024 and 1280 px and **Who changed what** on **Changes** at 1024 px (a long resource name or actor is cut, with the whole name on hover); a blocked route's operations on **Problems** break only after a dot; a change request whose blast radius was stored before 0.1.161 reads its specific reason alone, its folded coverage gap as *Coverage gap recorded before 0.1.161, not split by kind: …* and a bare *No impact* as the change management agent's note — on the change page and in the change request document, without rewriting the stored report; an RCA written after the upgrade heads its blast radius section with the change page's basis label (*Blast radius — Estimated impact*) instead of *Blast radius (certified)*, and one written before keeps its text; the dark theme draws native controls dark. The changes run in the control plane: private-network agents need no change. | `GET /healthz` reporting 0.1.162; the findings list, the runbook catalog and **Who changed what** fitting their card at 1280 and 1024 px without a sideways scroll |
| 0.1.161 | No schema change, no new route or permission. One new setting, `aiops_dashboard_pillars` (`dashboard_pillars` on `GET`/`PATCH /api/aiops/{tid}/guardrails`, `members.manage`; default `incident`, `compliance`, `change`); `GET /api/aiops/{tid}/dashboard/summary` carries `shown_pillars`, `shown` per ring and `samples` per mean. **Nothing else to do for the upgrade.** After it, the next estate harvest of each AWS read connection — the scheduled one (`aiops_inventory_interval_s`, daily by default), or **Harvest now** on the **Estate map** (every eligible connection) or on the connection in **Data sources** (`datasources.manage`; `POST /api/aiops/{tid}/xray/harvest`) — folds the duplicate Lambda function entries a map harvested before 0.1.161 holds (one `$LATEST` node per function) into the function's node; a function the map no longer holds keeps one node under its own name. A change request whose blast radius was stored before that harvest keeps it until its review or preview is opened again. **Behaviour that changes on upgrade:** the **Dashboard** shows three pillars — Incident & Problem, Compliance, Change Mgmt — until a workspace lists more under **Guardrails → Dashboard pillars**; the hub is one scalable picture; a ring and its **Pillar health** row read the same unresolved count (the column reads **Unresolved**); MTTA and MTTR print a comparison only from five or more incidents a side; list tables fit their card from 1024 px; the blast radius card and the change request document's 2.4 and 2.5 print one basis sentence and the map's coverage gaps by kind; the harvest keys a Lambda function's versions and aliases and a Step Functions state-machine version by the unqualified ARN (a Lambda layer version keeps its own node), so that first harvest also adds the `versions` and `aliases` properties to every function it reads and reports each such function once as changed — in the connection's harvest delta and as *changed externally* on the next Estate x-ray scan's **Estate changes since the last scan** card and the Estate map's **Changes since the last scan** mark; nothing changed in the account. The changes run in the control plane: private-network agents need no change. | `GET /healthz` reporting 0.1.161; the Dashboard showing three rings around the human gate at any window width, the **Unresolved** column equal to each ring's number; the **Dashboard pillars** card at the end of **Guardrails**; after the harvest, no **Estate map** object named `$LATEST` |
| 0.1.160 | Schema, additive: the partial index `ix_changerecord_tenant_cloud_write_executed` (`changerecord (tenant_id, executed_at)` over harvested cloud writes — the read every did-not-hold mark now makes inside alert intake), registered at start and built in the background while the app serves, like every registry index ([Schema changes after the pod is ready](/help/ops-upgrade-and-verify.md#schema-changes-after-the-pod-is-ready)); on a large change ledger the build takes longer, and until the index is valid that read is slower, not wrong — nothing to do first. One new route, `GET /api/aiops/{tid}/dashboard/summary` (`incidents.view`; the spend figures need `report.view`; MCP `dashboard_summary`); the incident list and summary accept `?status=unresolved` and the summary carries `resolved_by_day`; the Problems page reads the list's existing `?status=` filter; a recorded fix gains the state `recurred_after_outside_change` and its recurrence marks the field `outside_change_id`. No new setting, permission, audit action or chart value. **Nothing else to do for the upgrade.** **Behaviour that changes on upgrade:** `/portal/` opens the new **Dashboard** at `/portal/dashboard` (the former dashboard is replaced; a workspace with no enabled read connection still sees the Quick start once per browser session); a did-not-hold mark recorded after the upgrade names the recorded outside change that explains it and is not counted towards the ×N block, the candidate revocation or the not-good state — marks from before carry no such field and keep counting, existing problem rows are not rewritten. The changes run in the control plane: private-network agents need no change. | `GET /healthz` reporting 0.1.160; the sidebar's **Dashboard** opening `/portal/dashboard` with the six rings and a number that opens its list with the same total; the index `valid` in the `migrations` block of `GET /api/aiops/admin/overview` (or `indisvalid = true` on the database); a Problems row reading *recurred after an outside change ×N* beside its automation facts |
| 0.1.159 | No schema change, setting, route, permission, audit action or chart value change; one new field, `change_class_cause`, on a change's `meta` and on an incident's proposed change. **Nothing to do for the upgrade itself.** **Behaviour that changes on upgrade:** a Lambda runtime-upgrade change (`aws.lambda.update_runtime`, `aws.lambda.update_alias`) is a typed fix: its validation plan's rows and its post-fix recheck read the function's configuration and alias back on the step's own connection and land the verdict at $0 once its derived metric window closes, with no agent run — a change still `verifying` at the upgrade is graded that way on its next check, and a verify round reading *not read: no compiled check for a Lambda function* is no longer expected; a released or verified Lambda fix stamps only the finding of the function it wrote to. A change whose standard class dropped before the upgrade carries no cause and reads *runbook no longer backs it*; one that drops after it says why (`meta.change_class_cause`). The changes run in the control plane: private-network agents need no change. | `GET /healthz` reporting 0.1.159; a Lambda runtime-upgrade change's validation plan reading the function's configuration and alias as rows the platform read at $0; the runbook catalog's AWS-published rows reading *Published by AWS · …* |
| 0.1.158 | No schema change, setting, route, permission or chart value change; two new audit actions, `route.deviation_explained` and `route.asr_seeded`. **Nothing to do for the upgrade itself.** **Behaviour that changes on upgrade:** within one scheduler tick (a minute) of the start, every workspace with an enabled AWS write or read/write connection receives the five AWS-published step runbooks, disabled and not certified, bound to its oldest such connection (audit `route.asr_seeded`; a workspace that already holds one of the names keeps that route as it is and gets only the missing ones; a runbook a person deletes or renames afterwards is not added again) — a runbook catalog that lists *Not certified · disabled* rows it did not have before is expected, and nothing enables them; with `aiops_demote_on_deviation` on, a certified runbook route whose fix did not hold keeps its certification when the platform's own cloud audit harvest recorded a non-platform write on the same resource between the fix and the recurrence (`route.deviation_explained` on the recurrence incident; a run that fails its validation is still demoted), and a demotion recorded on 0.1.157 is not reversed — an approver certifies the route again; a scheduled release refused because the change class dropped lands with the full reason (what dropped, the approvals no longer count, review and release or schedule again). The changes run in the control plane: private-network agents need no change. | `GET /healthz` reporting 0.1.158; the `route.asr_seeded` audit row on each workspace with an AWS write connection and its five runbooks in the runbook catalog; the Approvals card of an automatically released change reading *Released automatically — Platform (pre-approved)* |

A version string from `/healthz` proves the control plane only. An older private-network agent ignores newer forwarded settings, so after every upgrade confirm the deployed agent version and one scoped datasource read.

### Webhook body cap {#webhook-body-cap}

From 0.1.149 the control plane answers `413` to a `/webhook/*` body over 1 MiB before it reads the body. Alertmanager, Grafana and the generic intake (`/webhook/alertmanager/*`, `/webhook/grafana/*`, `/webhook/generic/*`), which send a whole alert group in one notification, accept up to 8 MiB, and `/webhook/kubernetes-audit/*` up to 16 MiB. Earlier releases accepted any size where the load balancer in front sets no limit. A sender does not retry a `413`: Alertmanager and Grafana drop that notification, and every resend of the same group fails the same way. An alert is roughly 1 KB from Alertmanager and 2 to 3 KB from Grafana, so a grouped notification of a few thousand alerts can reach the cap. Each `413` is logged as `webhook '<source>': body ... bytes is over the ...-byte cap` (the source path segment only, never the workspace or token) and counted in `aiops_http_requests_total`.

Where a source sends larger bodies, raise the cap through `extraEnv` (bytes; `0` removes the cap; the 8 MiB and 16 MiB sources keep the larger of theirs and this one):

```yaml
extraEnv:
  - name: AIOPS_WEBHOOK_MAX_BODY_BYTES
    value: "16777216"
```

On the Alertmanager side, `max_alerts` on a `webhook_configs` entry keeps a notification under the cap by sending only the first that many alerts of a group; the rest are counted in `truncatedAlerts` and not delivered.

### Kubernetes audit webhook token {#kubernetes-audit-webhook-token}

From 0.1.156 `/webhook/kubernetes-audit/<workspace-slug>/<token>/<cluster-name>` takes only that cluster's own audit token, and the form without the cluster segment (`/webhook/kubernetes-audit/<workspace-slug>[?cluster=]`) answers `404`. Earlier releases accepted the workspace webhook token there, which every alert and change integration holds, and recorded what it posted as the platform's own audit trail. After the upgrade the API server's deliveries with the webhook token are refused (`401`) and the writes they carry are not recorded. For every self-managed cluster that posts its audit log:

1. Mint the cluster's token: `POST /api/aiops/{workspace}/k8s-audit-token/<cluster-name>` with `pillars.manage` (an `admin`-scope personal access token, or MCP `mint_k8s_audit_token`). The name is the last path segment the cluster already posts to. The token is shown once. The portal has no control for it: plan this step for someone who can call the API or MCP.
2. Put it in the audit webhook kubeconfig's `token:` and keep the `server:` URL ending in `/-/<cluster-name>`.
3. Restart the kube-apiserver and confirm a `200` on the next batch: the cluster's writes appear on the Changes page again.

EKS clusters read through `k8s_audit: true` on the AWS connection need nothing.

## Schema changes after the pod is ready {#schema-changes-after-the-pod-is-ready}

From 0.1.149, before it serves, the control plane only adds tables and columns: a new column carries its default in the catalog, so no row is rewritten, and every statement waits at most 3 seconds for a table lock (or meets a deadlock), then rolls back and retries, up to 5 tries; the fifth waits in line for up to 60 seconds, so queries on that one table can pause for that minute once. The schema step therefore takes about as long on a year of history as on an empty database. Startup as a whole does not yet: with leader election off, the control plane also recovers the runs a restart interrupted before it serves, and that grows with the number of runs that were running. The chart's startup probe allows 5 minutes (60 checks, 5 seconds apart) before liveness counts.

Indexes on tables that already hold rows, and one-off data fixes, run afterwards, in the background, while the app serves:

- one control-plane pod builds at a time (a Postgres advisory lock); a second pod waits and checks again every minute;
- each index is built with `CREATE INDEX CONCURRENTLY`: writes continue and nothing is locked against the API, the webhooks or the agents;
- each data fix runs in batches and records its progress as the platform setting `migration:<id>`, so a restart continues where it stopped and a finished fix never runs again.

To check one after an upgrade, on the database:

```sql
-- every registry index present and valid (false = a build that was interrupted)
SELECT c.relname, i.indisvalid
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname IN ('ix_incident_tenant_opened', 'ix_auditevent_action_target_at');
-- a build in progress, and how far it is
SELECT relid::regclass, phase, blocks_done, blocks_total FROM pg_stat_progress_create_index;
-- the data fixes: status running | done | failed, with the cursor
SELECT key, value FROM platformsetting WHERE key LIKE 'migration:%';
```

The full list is in Database schema. The control-plane log says `online schema: building <index>` and `online schema: built <index>` for each build, and `online schema: migration:<id> done` for each data fix.

An index left invalid (`indisvalid = false`: the pod was stopped mid-build, or the build was cancelled) is harmless: queries ignore it, and the builder drops and rebuilds it, after a minute when it failed while the pod runs (the wait doubles after each failure, up to an hour), at the next start otherwise. A data fix with `failed` is retried from its cursor the same way; its `error` field says why. The builder lifts `statement_timeout` and `lock_timeout` for its own connection, so a timeout set on the database role does not cancel a build.

To run the index builds yourself instead (a database whose changes a DBA makes by hand), set `AIOPS_DB_ONLINE_INDEX_BUILDS=false` through `extraEnv`. The control plane then logs, for every index still missing or invalid, the exact statement to run. Run them with a direct connection to the database: `CREATE INDEX CONCURRENTLY` cannot run inside a transaction, so it fails through a pooler in transaction mode.

### Schema work that finishes after the upgrade {#schema-work-that-finishes-after-the-upgrade}

Some schema work runs in the background once the control plane is ready, so it does not hold up the rollout: indexes built without locking writes, and one-shot data migrations. `GET /api/aiops/admin/overview` (platform superadmin) reports it in its `migrations` block:

- `indexes`: each background index with its `state`: `pending` (not started), `building` (with `progress` in percent and the build's `phase`, for example `waiting for old snapshots`: the build waits until every transaction older than it, on any table, has ended), `valid` (done) or `invalid` (a build that stopped; it is rebuilt);
- `one_shots`: each data migration with its `status`: `pending`, `running` (with `progress`, where it has got to), `done` or `failed` (with `error`);
- `pending`: how many are not done.

```bash
curl -s -H "Authorization: Bearer $SUPERADMIN_PAT" \
  https://aiops.example.com/api/aiops/admin/overview | jq .migrations
```

The upgrade is complete when `pending` is `0`. The metrics port exports the same as `aiops_migration_pending{name}`: `1` until that index or migration is done, then `0`. Keep the `migrations` block with the release record.

## Verify the release {#verify-the-release}

The following seven checks follow **Verify a release** in `/ops/docs`, Platform configuration. Checks 2–6 require a platform superadmin.

| Check | Pass condition |
|---|---|
| 1. `GET /healthz` | `ok: true` and the intended version: the image tag up to its first `-`, which is what the pipeline's own check compares; pipeline HTTPS/SNI certificate check passes |
| 2. Ops → Customers and Workspaces | Expected organizations and workspaces remain; their assignments are correct |
| 3. Ops → Overview → Agent runners | Expected execution mode and provisioner; active workspaces have live runners; no unexplained expired leases or exhausted retries |
| 4. Ops → Connector catalog | Expected connector types are present after startup sync |
| 5. Ops → Platform defaults | Saved model tier map survives; startup log confirms cache hydration if required |
| 6. Ops → Audit log | Fresh audit rows arrive |
| 7. Portal smoke | Incident opens, runner investigates and a human-reviewable remediation plan arrives |

Retain `deploy-log-<slug>`: its `deploy-log.json` records image tag, digest, chart, snapshot, host, ALB and commit. The app health endpoint reports its version. Add the tenant smoke incident URL and outcome to the same release record. A model list or login alone does not prove runner execution.

The supplied smoke script, `scripts/customer-smoke.sh`, proves the runner part of check 7 from a host that reaches the platform, as in [Install](/help/ops-install.md#verify-the-installation): an alert opens an incident and a runner investigates it on the real model. On an upgrade, always pass the workspace's current ingest token as `WEBHOOK_TOKEN`. With it unset the script mints a new webhook token, which replaces the workspace's token at once: every configured alert source then gets `401` until it is updated.

```bash
BASE=https://aiops.example.com TENANT=example-org \
  WEBHOOK_TOKEN="$(cat /secure/webhook-token.txt)" \
  ADMIN_PASSWORD_FILE=/secure/admin-password.txt scripts/customer-smoke.sh
```

The script signs in as `ADMIN_EMAIL` (default `admin@local`, the bootstrap superadmin), fires one generic alert for a new `aiops-smoke-<epoch>` service and waits up to `TIMEOUT_S` (default 1200 seconds) for the runner to finish the investigation. It fails unless a real model answered (per-model usage is recorded), the run booked a cost above zero and the RCA is longer than 50 characters; on success it prints `SMOKE OK` and the incident URL. It reads the investigation as that account, so the account must still be a member of the workspace: an `HTTP 404` on that read means it is not. A `TENANT` slug that does not exist yet creates a new workspace, and workspaces cannot be deleted, so check the slug. The smoke costs a real model call, does not execute the proposed fix, and leaves its synthetic incident open for review.

Published documentation is part of the release gate. On every push to `main` the test pipeline (`.azuredevops/azure-pipelines.yml`, step *docs leak scan*) runs `scripts/docs_leak_scan.py` over everything the help site publishes — the product chapters, the repository docs outside the vendor-only folders and the film transcripts — and fails on any customer, person or host name. The forbidden-name list is the secret pipeline variable `DOCS_LEAK_PATTERNS` and exists nowhere else: not in the repository, not in the image. The scan logs only file, line and pattern index, never the matched text, and a run without the secret logs a visible warning instead of a pass.

## Signature and immutable-tag checks {#signature-and-immutable-tag-checks}

Signing is conditional. The release pipeline signs only when its signing key is configured; the deployment pipeline verifies only on that configured path. A warning about unsigned/unverified images is not successful signature verification. The published security review records signing as off; retain the actual pipeline result when that changes.

Registry image tags and OCI chart versions are immutable. The first build publishes the plain version tag and chart. For a rebuilt image of the same version use its full build tag; a chart change requires a new version. The chart currently pulls by tag even though the pipeline records the digest; digest pinning and cluster admission verification are not delivered.

Do not improvise a signing command or install a new trust key during an upgrade. Use the configured verification stage and retain its receipt. For a failed release follow [Rollback](/help/ops-roll-back-a-release.md), including the secret-store rotation boundary.

Sources: [Deployment runbook §3](/help/ops-installation-runbook.md#3-upgrade), [Availability](/help/ops-availability.md), [Platform configuration, Verify a release](/help/ops-platform.md#verify-a-release), and Security review AR-11, AR-12 and AR-19.


## Next steps

- Next: [Post-upgrade checks, 0.1.109 to 0.1.143](/help/ops-upgrade-checks-0-1-109-to-0-1-143.md)
- [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt)
- [Documentation home](/help/)