Upgrade and release verification

The deployment pipeline is the normal upgrade path.

How-to · Updated 2026-10-28 · Describes release 0.1.164

Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).

The deployment pipeline is the normal upgrade path. The current operating baseline is 0.1.130 (image tag 0.1.130, chart charts/aiops:0.1.130, tag v0.1.130); select an image and chart from a published release record, not from an assumed newest tag.

Prepare and approve

  1. Read Release notes, including policy, schema and key changes. Book a maintenance window: the single control-plane replica uses Recreate, with about a minute of planned downtime.

  2. Merge the released tag into customer/<slug> or rebase that branch on the release. The pipeline runs the files on that branch; stale files mean stale deployment steps. Historical runbook §3 example below; select the approved target from its release manifest before running it:

    git merge v0.1.130

  3. Queue deploy-customer.yml from that customer branch with customer set to the installation slug, imageTag set to the approved tag, chartVersion set to the matching chart and bootstrapDb=false.

  4. Read every unit in Plan before approving aiops-<slug>. An application-only upgrade normally shows No changes, with at most output-only changes. Adopting the any-account IAM policy below intentionally changes IAM; review that change. Unexpected infrastructure changes need review before approval.

  5. The Deploy stage applies infrastructure, resolves and records the image digest and chart, checks signing when enabled, takes and waits for an RDS snapshot, and performs helm upgrade --install --atomic --wait.

Adopt any-account keyless AWS on an existing install

Merging a release does not change the files render produced: envs/<slug>/ and deploy/customers/<slug>/values.yaml. An installation rendered before 0.1.109 retains its broad estate-role assumption policy and has no AIOPS_KEYLESS_AWS_ANY_ACCOUNT flag in its rendered values.

Before adopting this mode, confirm every keyless estate role outside the platform account is named aiops-* without an IAM path; other targets become unreachable after the policy is narrowed. Bring envs/<slug>/iam/tf/main.tf's assume_estate policy up to the released customer template, and add AIOPS_KEYLESS_AWS_ANY_ACCOUNT: "true" to the rendered values. Review the expected IAM policy change in that upgrade's Plan. The platform's own account still uses a staff-prepared pair. This is an upgrade procedure, not a claim that the rehearsal has adopted the new mode.

The customer pipeline runs a fresh apply after approval; it does not bind approval to a saved plan. Keep the branch stable through the run and review the final apply log. This accepted limitation is recorded as AR-19 in the Security review.

Post-upgrade checks by release

Read every release between the installed version and the target. The release notes are the authority; this table lists the operator follow-up each release asks for and the evidence to keep with the release record. Three versions are never installed: 0.1.106 (void: its images fail at start), 0.1.115 (the control plane ran out of memory on a long-harvested estate; fixed in 0.1.116) and 0.1.127 (a credential named client_token could appear in clear; fixed in 0.1.128). 0.1.107 was superseded the same day by 0.1.108. A version whose build fails is void and is never retagged.

Release crossed Operator follow-up Evidence to retain
0.1.109 to 0.1.143 An install older than 0.1.144 crosses these first: their rows are in Post-upgrade checks, 0.1.109 to 0.1.143, oldest first, then continue here. as that page lists per release
0.1.144 Schema, additive and created at start: new table compliancereport (one row per workspace per UTC day) and the JSON column investigation.triage. Re-run the AWS CloudShell setup for every account it set up (a role from the deploy/aws module: re-apply it). The read role gains config:DescribeConformancePacks, config:DescribeConformancePackCompliance, ssm:DescribeInstanceInformation, ssm:DescribeInstancePatchStates, backup:ListBackupJobs, ec2:DescribeVpcEndpoints, ec2:DescribeTransitGateways, ec2:DescribeTransitGatewayAttachments, ec2:DescribeTransitGatewayVpcAttachments, ec2:DescribeTransitGatewayRouteTables and ec2:SearchTransitGatewayRoutes (a role on ReadOnlyAccess already has them). The write role gains ec2:CreateImage, ec2:CreateSnapshot and ec2:CreateReplaceRootVolumeTask on instances and volumes tagged aiops-managed=true, the same three plus ec2:CreateTags on images, snapshots and restore tasks created with the aiops:change tag, and ec2:DescribeImages, ec2:DescribeSnapshots and ec2:DescribeReplaceRootVolumeTasks; nothing that deletes. Re-run the Azure Cloud Shell setup for a write connection: its role gains Microsoft.Compute/disks/read, Microsoft.Compute/snapshots/read and Microsoft.Compute/snapshots/write. The guided read role also gains health:DescribeEvents for the automated triage's cloud-health step (a role on ReadOnlyAccess already has it); until the setup is re-run, that step reads not available. Until the setups are re-run, Scores lists the new reads as denied and a reboot stops at its backup step (UnauthorizedOperation) before anything changes. Defaults that change behaviour on upgrade: a backup before every reboot-type step (AIOPS_BACKUP_BEFORE_CHANGE, on; kept 7 days by tag, never deleted by the platform; a reboot-type plan over a CLI write connection is refused at review); a certified step runbook carries its own operations (AIOPS_RUNBOOK_OWNS_OPERATIONS, on): Pre-approved operations covers single-call routes only, a fetched skill alone no longer makes a change standard, and pre-approving a fix waits until its incident is resolved; a person may only raise a change class, and a P1 a certified runbook covers is released under the standard rules (requests recorded before keep their stamped rules); denied access attempts open SecOps incidents while the SecOps agent is enabled (AIOPS_DENIED_ACCESS_TO_SECOPS, on); a scanner HIGH maps to error (P2) instead of critical (P1) (register rows change on the scanner's next send, open incidents keep their priority); a P3/P4 finding with a prepared fix waits for Analyse; the automated triage reads cloud health and known issues before each incident analysis and adds ## Fault domain to its report (AIOPS_TRIAGE_CHECKLIST, on); the compliance reports are read once a day (AIOPS_COMPLIANCE_REPORT_INTERVAL_S = 86400). Off until a workspace turns them on: unauthorized-change incidents (AIOPS_UNAUTHORIZED_CHANGE_INCIDENTS; fill change_authorized_principals first), several runbooks per change (AIOPS_RUNBOOK_MULTI_COVER), the change management agent (AIOPS_CHANGE_AGENT_CR) and its fix drafting (AIOPS_CHANGE_AGENT_OWNS_PLAN); AIOPS_PATCH_WITHIN_DAYS is 0 (no rule). A private-network agent in database mode needs this release for the triage steps on its runs (in API mode the control plane does them); turning on the change management agent's fix drafting needs every private-network agent on this release. The re-run setup output; the first Scores report with no denied call; GET /guardrails switch_state per workspace; one incident's Automated triage card
0.1.145 No schema change. Re-run the AWS CloudShell setup for every write connection it set up: the write role gains iam:SimulatePrincipalPolicy on its own ARN only, with an explicit deny on every other principal (a role from the deploy/aws module is unchanged and its permission check keeps reading unknown). Until the check can answer, the step-runbook automatic lane does not run a runbook on its own; a person releases it. Behaviour that changes on upgrade: an unconfirmed permission check stops an automatic step-runbook run; a change request the platform executed stops covering outside writes 60 seconds after its execution; STS calls are no longer harvested as changes; a fix whose alerts cleared at the source and whose validation plan reads as expected clears with no agent run; the agreed comparison table's sha256 changes (A.8.8 and A.8.13 notes), so a recorded acceptance reads as changed since and needs a new acceptance; a reviewed revert that re-creates a security-group rule with a description needs a new review (its digest now carries the description). The re-run setup output; a step runbook's Check runbook showing the permission check passed; GET /guardrails switch_state per workspace
0.1.146 No schema change. Re-run the AWS CloudShell setup for every account it set up (a role from the deploy/aws module: re-apply it). The read role gains lambda:ListFunctions, ecr:DescribeRepositories, ecr:DescribeImages and ecr:ListTagsForResource (a role on ReadOnlyAccess already has them); until the setup is re-run, the harvest's Coverage lists lambda:ListFunctions as denied. The write role gains ssm:SendCommand with the AWS-RunPatchBaseline document only on instances tagged aiops-managed=true, ssm:GetCommandInvocation, ssm:ListCommandInvocations and ssm:DescribeInstanceInformation, and Reachability Analyzer (ec2:CreateNetworkInsightsPath and ec2:StartNetworkInsightsAnalysis only with the aiops:change tag on create, ec2:StartNetworkInsightsAnalysis on a path and ec2:DeleteNetworkInsightsPath and ec2:DeleteNetworkInsightsAnalysis only on what carries it, account-wide ec2:Describe* (instance and launch-template user data included), ec2:GetManagedPrefixListEntries, ec2:GetTransitGatewayRouteTablePropagations, ec2:SearchTransitGatewayRoutes, elasticloadbalancing:Describe*, network-firewall:Describe* / List* and tiros:CreateQuery, ExtendQuery, GetQueryAnswer, GetQueryExplanation, GetQueryExtensionAccounts); nothing uses those yet. ssm:GetCommandInvocation and ListCommandInvocations are on * and return any command's output in the region, and AWS-RunPatchBaseline takes a BaselineOverride and an InstallOverrideList IAM cannot restrict, so the write role can install packages on a tagged instance. The backup's ec2:CreateTags is now bound to the backup's own create calls (ec2:CreateAction = CreateImage, CreateSnapshot, CreateReplaceRootVolumeTask) and to the tag keys aiops:change and aiops:retain-until. A deploy/aws write role also gains iam:SimulatePrincipalPolicy on its own ARN and a region pin on its aiops:change-scoped statements. Defaults that change behaviour on upgrade: the scheduled Security Hub / Defender pull is on every 15 minutes (AIOPS_SCANNER_SYNC_INTERVAL_S = 900; a workspace that stored 0 stays off), whatever the SOC 2 / ISO 27001 switch says, and its Inspector CVE read is on unless a connection says pull_vulnerabilities: false. With Analyse P1/P2 findings automatically on, every HIGH or CRITICAL CVE starts one paid analysis (measured USD 1.04 to 2.92 each): before the first pull, set each workspace's daily_cost_limit_usd, set Scanner sync to off where the pull is not wanted, and give an AWS read connection into a shared account Only findings on resources tagged (finding_tag_scope, one key=value; any other value fails that connection with invalid_tag_scope, it never pulls wider). The change management agent drafts every change request (AIOPS_CHANGE_AGENT_CR, on): one change-agent run of at most 8 steps per change request, about USD 0.33 measured. A workspace that set aiops_change_agent_owns_plan: true without setting aiops_change_agent_cr now gets Phase 2 (the agent drafts the plan); set aiops_change_agent_cr or aiops_change_agent_owns_plan explicitly before the upgrade if that is not wanted. Screens and names: the change page's "Verification" card is Validation plan and "Safety net" is Rollback plan; "Alert correlation" is Service correlation in the sidebar and on L2 (skill id alert-correlation, the /alerts route and the unmapped filter are unchanged); the first firewall harvest lists deny rules as added once. API and MCP: the automated triage's triage_checklist[*].step numbers follow the new order (1 dependency map, 2 known issues, 3 cloud health, 4 recent changes, 5 signals, 6 summary, 7 fault domain); match a step by its title, not its number. The re-run setup output; the next harvest's Coverage with no denied lambda:ListFunctions; a step runbook's Check runbook permission check; GET /findings/sync (interval_s, interval_inherited) and GET /guardrails switch_state per workspace
0.1.147 Schema, additive and created at start: new table syntheticcheck, the columns incident.merged_into_id and investigation.scope_deferred (JSON), and the index ix_incident_tenant_opened on incident (tenant_id, opened_at). No new permission. New routes: POST /api/aiops/{tenant_id}/changes/{change_id}/reject (approvals.decide), GET and POST /api/aiops/{tenant_id}/monitors, PATCH and DELETE …/monitors/{check_id}, POST …/monitors/{check_id}/run and POST …/monitors/derive (datasources.view / datasources.manage), with the MCP tools reject_change, list_monitors, create_monitor, update_monitor, delete_monitor, run_monitor and derive_monitors. The storm gate is removed: aiops_storm_threshold and aiops_storm_window_s (a workspace setting, or AIOPS_STORM_THRESHOLD / AIOPS_STORM_WINDOW_S on the control plane) are accepted and ignored, and a burst of unrelated services now opens one incident each (the shared-alert-name fold, aiops_fold_shared_cause_threshold 3, stays). Grouping reads the new windows aiops_storm_group_window_s (300 s, at most 3600) and aiops_storm_spread_window_s (600 s, at most 7200), whole seconds, platform AIOPS_STORM_GROUP_WINDOW_S / AIOPS_STORM_SPREAD_WINDOW_S or a workspace setting; aiops_storm_agent keeps its meaning (the grouping agent, on). A __storm__ incident left open is not split any more: no alert joins it, it resolves when its alerts clear or a person resolves it, and an alert on one of its services opens that service's own incident beside it. A dashboard or alert on aiops_webhooks_total{outcome="storm"} must read outcome="grouped". SOC 2 / ISO 27001 are off in every workspace (AIOPS_COMPLIANCE_FRAMEWORKS, off): the first scheduler tick after the upgrade resolves every open platform-check finding (control:<id>, by: scope-change; one a person holds stays), and a compliance scan answers 409. AIOPS_COMPLIANCE_FRAMEWORKS is platform-wide: true (helm extraEnv) turns SOC 2 / ISO 27001, EST-01 to EST-06 and the platform alarms back on in every workspace whose compliance.enabled is not an explicit false, customer workspaces included. Set it only once the customer has agreed the comparison table, never for a Test Lab twin on a shared control plane; Test Lab cases that expect the platform's own checks get no scan while it is off. Defaults that change behaviour on upgrade: the known-issue fast path (AIOPS_KNOWN_ISSUE_FAST_PATH, on: an alert-started analysis rated 100 % relevant runs capped at 3 steps); post-change validation in 3 rounds 300 s apart (a rule's minutes no longer times anything: validation_rounds: 1 with validation_interval_s = minutes × 60 reads about the old single window); validation signals derived from the change request's plan when no rule matches (AIOPS_VALIDATION_DERIVED, on); a 10-minute hold on the automatic verify after a start, reboot or restart; daily synthetic-check suggestions (AIOPS_MONITOR_DERIVE, on): up to 20 internet-facing load balancers and API Gateway APIs per workspace are probed from the control plane, GET / every 300 s, in trial (nothing opens) until a person keeps or discards them; an API Gateway call is billed and can invoke its integration, so turn aiops_monitor_derive off where that is unwanted; backup findings from the daily AWS Backup read (AIOPS_BACKUP_FINDINGS, on: one P3 incident per workspace while SOC 2 / ISO 27001 are off); CI/CD service-account writes recorded as changes (AIOPS_K8S_CI_NAMESPACES: flux-system, jenkins, gitlab-runner, tekton); Azure DevOps workitem.updated deliveries that change the state are signals, so a service hook that already sends them now resolves incidents on Done, Closed, Resolved or Removed, and a state change on a work item the platform has not seen opens an incident with an analysis (any other update is ignored; an incoming ticket missing required fields is not held yet); AWS Config findings key on their own rule (control:<rule-name>): a finding already open in the shared incident stays there until it clears, new ones open one incident per rule. Roll private-network agents to this release: an agent in database mode needs it for the service's resources in the prompt, a run's resource selection and the vulnerable-package plan per resource type. GET /guardrails per workspace: switch_state.known_issue_fast_path, verification.validation_derived and compliance_frameworks_available: false; the platform-check findings resolved by: scope-change; the first incident.grouped audit row; Synthetic checks after the first daily suggestion; agent version
0.1.148 No schema change or permission. New workspace setting aiops_grouping_wait_s (whole seconds 0–600, default 300, 0 = no grouping wait; PATCH /api/aiops/{tenant_id}), separate from aiops_storm_group_window_s: a workspace that raised the group window keeps a 300 s wait. New audit action change.approval_lapsed. New route POST /api/aiops/{tenant_id}/changes/{change_id}/close-without-verdict (approvals.decide, body {"reason": "…"}; 409 until the change has gone 24 hours without a verdict), with the MCP tool close_change_without_verdict. An approved change released by a person who did not vote runs again; a change the 0.1.147 dispatch re-check set to rejected stays rejected, so its incident needs a new change request. Behaviour that changes on upgrade: an incident on a service that depends, over a dependency allowed to fold, on another service holds its analysis for the grouping wait (aiops_grouping_wait_s, 300 s) while its ticket and notification go out as usual, and an alert that clears inside the wait resolves its incident with no analysis (when it comes back it is analysed at once); with a reopen window set, an incident a person closed as known noise is no longer reopened (the next alert on its key opens a new incident, analysed as usual), and a caller whose resolved incident would reopen joins its root's open incident instead. One-shot clean-up of the connection tag scope (Only findings on resources tagged): AWS Backup records already open for a resource outside it clear at the next daily read of their region (one compliance.backup_out_of_scope audit row; a tag none of the resources read carries clears nothing and the Backup card says so); suggested synthetic checks nobody kept whose resource is outside it are removed at the next daily suggestion run, also where suggestions are turned off (named in the monitor.derived row); built-in detection pairs already in alarm on such a resource resolve (outside the workspace scope or the connection's tag scope), except under a scope that cannot be read or that none of the connection's resources carries, which resolves nothing. Security Hub findings of DynamoDB.1, the deletion-protection controls, ElastiCache.1/.2, FSx.1/.2 and Redshift.3 move family when Security Hub next sends them (daily). The rollback sweep no longer escalates a change that stopped at its backup. The fixes run in the control plane; roll private-network agents as usual. An approved change released by a second approver reaching change.executed; a dispatch refusal's change.release_refused row naming its reason; the first compliance.backup_out_of_scope row and the Backup card's count; a caller incident reading Held for triage — grouping window: …; agent version
0.1.149 Image and chart bump, no values change; no new table or column. The start's schema step no longer scans tables: each statement waits at most 3 s for a table lock and retries (5 tries, the last waiting up to 60 s); with leader election off the start still resumes the runs the restart interrupted before it serves. Once the app serves, the control plane builds seven new indexes with CREATE INDEX CONCURRENTLY (ix_auditevent_action_target_at, ix_auditevent_tenant_target_at, ix_incident_tenant_key_status, ix_incident_tenant_status_resolved, ix_investigation_tenant_started, ix_knowledgeitem_tenant_name_version, ix_changerecord_live_status; minutes on a large audit trail, writes continue) and runs migration:null-defaults-v1 once (Schema changes after the pod is ready); both need a direct database connection, not a pooler in transaction mode. A workspace without a PII salt gets one at the first start. The chart adds a startup probe (60 × 5 s) before liveness counts. A webhook body over 1 MiB is answered 413 (Alertmanager, Grafana and the generic intake: 8 MiB; kubernetes-audit: 16 MiB) and the sender does not retry it: where a source posts larger groups, raise AIOPS_WEBHOOK_MAX_BODY_BYTES through extraEnv or set Alertmanager's max_alerts (Webhook body cap). aiops_change_records exports open statuses only: a dashboard or alert on completed, rolled_back, rejected or recorded reads nothing. New series: aiops_scheduler_subtick_seconds, aiops_migration_pending. Rollback: the previous image with this chart kept (Rollback); nothing in the database is undone and the new indexes stay. The migrations block of GET /api/aiops/admin/overview with pending: 0 (every index valid, null-defaults-v1 done); /healthz version; the pod's time from start to Ready (under 30 s when no run was interrupted); every 413 line in the control-plane log, with its source
0.1.150 No schema change, setting, route or permission. Behaviour that changes on upgrade: a step runbook's or the analysis's fix with a step on a resource that only an alert merged, folded or grouped into the incident names (and none on the incident's own) is never a standard change, and its release is refused, by a person (403, change.release_refused or Fix failed reading refused: …) or automatically (route.match_skipped, reason target_not_own, also for a direct API route whose fixed target is such a resource); a change request drafted before the upgrade keeps its class, but its release is refused the same way: re-analyse that incident. A clear of an incident whose executed change acted only on such a resource completes the change with the verdict uncertain and no last-known-good, memory or runbook credit, and writes incident.fix_not_credited; a change credited before the upgrade keeps its verdict. A validation round is clean only when every one of the incident's own alarms passed: an own alarm that read no data fails the round when it treats missing data as breaching and CloudWatch reads it in ALARM (the read role's existing cloudwatch:DescribeAlarms), and otherwise leaves the round not observed (an alarm that treats missing data as not breaching reads OK); a window opened before the upgrade, whose alarms all count as the incident's own, reads a round with a silent alarm as not observed instead of clean, and such rounds hand the fix to the agent verify. An incident alarm on a count that publishes nothing while healthy (errors, 5xx) and keeps CloudWatch's default TreatMissingData (missing) now leaves its rounds not observed: set it to notBreaching. An approval and the run of a change are now two ledger entries (approved, then released by the person who ran it); entries written before the upgrade are unchanged. A reopen chooses the suspected change again (incident.suspect_repinned, trigger reopen). Roll private-network agents in database mode: they draft the change request, its class and scope. On an incident holding a merged-in alert, a fix on the merged-in resource: its class reason and its review naming that resource, and no release offered; the next validation round listing every V-item; the change request document of a two-person release listing Approved by and Released by; agent version
0.1.151 No schema change, setting, route or permission. Behaviour that changes on upgrade: an incident's fix estimates its blast radius with the incident's service in every review (with a harvested estate, a step the estimate cannot model exactly can also name the one resource carrying that name), so an approval recorded on the incident page before the upgrade for such a fix no longer matches its digest and is asked for again; nothing ran under it. Once a step of a released change has run, the check before each later step keeps the blast radius the release was approved on. A certified step runbook's automatic release that ran no step, or stopped before a later one, writes incident.auto_remediate with not_dispatched or stopped_before and reason; rows written before the upgrade keep reading a step failed. Every incident's error start, ranking resource, alarm read, RCA Timeline start and triage resources now come from the alert that opened its current episode among its own alerts, and the platform's own fix for the incident is never ranked, so the Probable cause card of an incident that absorbed another service's incident can rank differently on its next read. A reopen stamps previous_end, previous_from and resolution_sig on incident.recurrence: an incident reopened after the upgrade is ranked from where its previous episode ended and drops the earlier episode's resolution note at its next close; one reopened before the upgrade has no stamp: its window reaches back to the previous error on its key or the last verified fix, else the look-back cap, and it keeps that note. A validation window still running at the upgrade also gets the added round, and the scheduled check takes a remediating incident before the other incidents of its workspace. Model calls go over httpx instead of litellm's aiohttp transport: HTTPS_PROXY and NO_PROXY still apply, and NO_PROXY entries are matched by httpx's rules. The fixes run in the control plane; roll private-network agents as usual: an agent's model calls move to httpx once it runs this release, and an agent in database mode needs it for the triage's own-alert resources on its runs. The next automatic step-runbook release reaching change.executed, or its Timeline row reading Automatic fix refused before dispatch with the reason; cause.window.floor_basis previous_episode with floor_incident_id the incident itself in GET /api/aiops/{tenant_id}/incidents/{id}/lkg of an incident reopened after the upgrade; no Unclosed client session line in the control-plane log; agent version
0.1.152 No schema change, setting, route or permission. Behaviour that changes on upgrade: a change harvested since a verified fix on the resource it touched now counts against the known-issue relevancy whether the ledger names the resource by its bare EC2 id or its ARN, and a harvested change counts for an incident by its resource or its linked services, never by its event source: an incident whose fixed instance or volume changed since the fix reads 85 % and its analysis runs in full where it took the fast path before. The same match widens the record's changes on these resources since that resolve on the incident page and in the analysis's PRIOR RESOLUTION. The relevancy recorded on a triage gains fix (held, did_not_hold) and the fast-path instruction says whether the prior fix held; a run started before the upgrade and restarted after it says only a verified fix. Triage step 2, the header chip and the incident list read recurring ×N / recurring on a new incident of a resolved alert; recurrence.kind stays reopened, and the incident detail's recurrence gains kind_at (an in-place reopen). The Probable cause card shows the window's own look-back sentence. The fixes run in the control plane; roll private-network agents as usual: an agent in database mode computes the relevancy on its runs. On the next new incident of a resolved alert, the header chip reading recurring ×N; investigation.triage.known_issues.relevancy.fix in GET /api/aiops/{tenant_id}/investigations/{id} of its triage run; agent version
0.1.153 Schema, additive and created at start: the JSON columns detectionstatus.rule_since and detectionstatus.baselines. No new route. Before upgrading, ticket intake: a Jira, ServiceNow or Azure DevOps ticket sent in must now name service, severity, priority, assignee and notify (AIOPS_TICKET_REQUIRED_FIELDS, a JSON list, default all five; workspace ticket_required_fields, [] = none of the five — a ticket naming no service still waits while aiops_unmapped_triage is on, the default); one missing any waits in the triage queue (no analysis, notification or automatic fix) and AIOps asks on it once through the workspace's Ticket system route. Update the ServiceNow inbound business rule to the new template (it posts severity, impact, urgency, assignee, assignment group and watch list, and fires on Update as well as Insert), name the severity and who to notify on Jira issues (Jira has no field for either: Severity: / Notify: line or severity: / notify: label), tag Azure DevOps work items service:<name> and notify:<team> (or the same description lines), and enable Jira jira:issue_updated and Azure DevOps workitem.updated events so an update can move a held ticket on — or narrow ticket_required_fields for each workspace first. Re-run the AWS CloudShell setup for every account it set up (a role from the deploy/aws module: re-apply it): the read role gains rds:DescribePendingMaintenanceActions (built-in detection R36; a role on ReadOnlyAccess already has it), and the write role gains ec2:ModifyNetworkInterfaceAttribute on network interfaces tagged aiops-managed=true and on any security group in the region, for the EC2 patch runbook's isolation and restore (the group side cannot be narrowed by tag, so the role can attach any group in the region, an open one included, and the same action changes other interface attributes such as SourceDestCheck), and ssm:DescribeInstancePatchStates. Tag the instances' network interfaces aiops-managed=true too. Until the setup is re-run, an isolation stops at its first interface with UnauthorizedOperation, nothing changed. New setting aiops_patch_findings (AIOPS_PATCH_FINDINGS, on; a workspace override): the daily Patch Manager read lands each non-compliant node in the tag scope as a Patching finding (control PATCH.COMPLIANCE, one incident per workspace; HIGH when a critical patch is missing, which is P2, else MEDIUM, P3), unless Security Hub's SSM.2 already holds that node open. With Analyse P1/P2 findings automatically on, the first read after the upgrade can start one paid analysis per workspace, and the incident's ticket and notification go out on open; set aiops_patch_findings: false before the upgrade where that is not wanted. The demo workspace gains the disabled example route EXAMPLE: EC2 patch with backup and isolation on a disabled placeholder write connection example-aws-writer; other workspaces create the route from the request in the runbook chapter, named remediate-…. New runbook operations, used only by a runbook that carries them: aws.ssm.patch_baseline, aws.ec2.isolate and aws.ec2.restore_sgs. Built-in detection gains R26–R37 (latency and fault baselines, EKS, Kubernetes pod and node state through a read-only kubectl connection, EC2 and RDS scheduled maintenance at P4 (severity info), AWS Health issues the estate is exposed to at P2): on a workspace already armed each new rule watches in shadow for the shadow period (24 h by default) from the first poll it covers something, then arms itself; Arm now arms them at the next poll. A built-in error alert (R37) is analysed, ticketed and notified like a critical one. New audit actions ticket.info_requested, ticket.info_received, change.isolation_left and compliance.patch_out_of_scope; new metrics aiops_detection_episodes_total and aiops_detection_reads_total, and op="ask" on the ticket metric. The changes run in the control plane; roll private-network agents as usual. The re-run setup output; a test ticket with a field missing waiting in the triage queue with its question on the ticket; the detection card's per-rule shadow line; the next daily Patching card and its findings; agent version
0.1.154 No schema change, no new route. Re-run the AWS CloudShell setup for every write connection it set up (a role from the deploy/aws module: re-apply it): the write role gains codebuild:StartBuild and codebuild:BatchGetBuilds on the one CodeBuild project aiops-image-rebuild in the connection's region while it carries aiops-managed=true, ecr:DescribeImages, and explicit denies of the StartBuild overrides that change what the build runs (buildspec, source, secondary sources, build image, compute, environment type, privileged mode, fleet, and any variable but the five the platform sends), for the container image rebuild. The project is the customer's: create it with deploy/aws image_rebuild = true, tag each repository it may push to aiops-managed=true and make those repositories' tags immutable. Until the setup is re-run, a rebuild stops at its start with AccessDeniedException before anything is built. The opt-in pull block (aiops_ecr_block_pulls, Allow a runbook to block pulls of a vulnerable image, off) also needs ecr:GetRepositoryPolicy, ecr:SetRepositoryPolicy and ecr:DeleteRepositoryPolicy, which only deploy/aws with image_pull_block = true grants. With image_pull_block = true, more than six regions (fewer with a tag key longer than aiops-managed) put the deploy/aws write policy over IAM's 10,240-character limit and the apply fails. A write role set up before 0.1.146 also lacks the Reachability Analyzer statements the neighbour reachability check uses (aiops_reachability_checks, API only, off; at most aiops_reachability_max_per_change analyses a change, 10, about USD 0.10 each billed to the AWS account). Both switches stay off until a workspace turns them on. New setting aiops_image_reminder_h (AIOPS_IMAGE_REMINDER_H, 24, 0–720), new notification event image_not_deployed (a target that lists its events gets it only once ticked; it repeats until the scanner stops reporting the old digest — moving the workloads alone does not stop it: delete or untag the old digest once nothing runs it), new audit action image.not_deployed, new runbook operations aws.ecr.rebuild_image and aws.ecr.block_image_pulls. Behaviour that changes on upgrade: the step-runbook permission check reads an action the write role lacks as missing — IAM's list of condition keys on a deny, implicit or explicit, no longer makes it unknown — so certification is refused where it went through before; a route certified before the upgrade keeps its certification, but its next Check the write connection's permissions and its re-certification read missing, and so does the check before an automatic run, except on the instance's network interfaces, whose tags it does not read (unknown: a person releases the change). A deploy/aws role scoped by a tag other than aiops-managed=true reads missing at certification too, until the write connection's Scope JSON names that tag: paste scope from the re-applied module's datasource_snippet, which now carries resource_tag. The changes run in the control plane; roll private-network agents as usual. The re-run setup output; Check the write connection's permissions on each certified step runbook reading every permission is granted; agent version
0.1.155 Schema, additive and created at start: the partial index ix_changerecord_live_status_v2 (not_needed is a new terminal change state; the 0.1.149 index stays). No new setting or route. Re-run the AWS CloudShell setup for every write connection it set up (a role from the deploy/aws module: re-apply it). The write role's statements no longer fit IAM's 10,240-character limit for a role's inline policies, so the backup and restore, Reachability, image rebuild and Lambda runtime statements move into four customer-managed policies attached to the write role only; the inline policy keeps the rest and the Deny boundary, and no statement's scope changes. The re-run's change set adds the four policies and modifies the write role: the stack update creates them, there is nothing to delete first, and the setup's review accepts only the template's own roles and policies, added or modified, never replaced or removed. Whoever runs the re-run (or applies deploy/aws) now also needs iam:CreatePolicy, iam:GetPolicy and iam:AttachRolePolicy, and iam:CreatePolicyVersion and iam:DeletePolicyVersion once a later release changes a policy (Terraform also reads iam:GetPolicyVersion and iam:ListPolicyVersions); an installer scoped to roles only sees the stack update fail and roll back, the role unchanged. deploy/aws creates <name_prefix>-write-backup, -reachability, -image-rebuild and -lambda-runtime and attaches them (four of the ten managed policies IAM attaches to a role by default); its write policies now fit every commercial region, with or without image_pull_block. The write role gains: the restore of an image backup — ec2:CreateReplaceRootVolumeTask from this account's own image or snapshot carrying aiops:change, into a new root volume only, its task tagged on create (0.1.144–0.1.154 allowed the image only with a request tag an existing image never carries, so AWS refused Roll back of the EC2 patch runbook; until the setup is re-run it still does, with UnauthorizedOperation before anything changes, and Check the write connection's permissions, the check before an automatic run and re-certification of the patch runbook read ec2:CreateReplaceRootVolumeTask missing); ec2:CreateNetworkInsightsPath on the network interfaces in the region (until re-run every neighbour reachability row reads not observed with UnauthorizedOperation); and lambda:PublishVersion, UpdateFunctionConfiguration, UpdateAlias, GetFunctionConfiguration and GetAlias on functions tagged aiops-managed=true (the Lambda runtime upgrade; until re-run it stops at its backup with AccessDeniedException, nothing changed). A CodeBuild project aiops-image-rebuild created by deploy/aws before 0.1.155 runs the old buildspec, which refuses a source package such as expat, until deploy/aws is re-applied. Switches and defaults: no setting or default changes. With both automatic-fix switches on, a compliance control, scanner finding, vulnerability or security alert now waits for a person on every lane, a certified step runbook's included. aiops_reachability_checks (off) also adds the reachability permissions to a runbook's permission check when on. The demo workspace gains the disabled example route EXAMPLE: Lambda runtime upgrade with version backup. A Jira or Azure DevOps ticket route fills the component, assignee or priority only after Check is run on it and the route saved again. Send test ticket works on a disabled route (it answered 409) and never enables it. Behaviour that changes on upgrade: a step runbook's change request is drafted at the tier its release needs (L2 for its API steps, L3 with a pull block) instead of L1; the change request document prints the runbook's own rollback steps; the patch runbook's post-change check reads the patch command's status and Patch Manager's missing and failed counts, and the restored groups against those the isolation recorded, instead of the instance's power state (a read that cannot answer is not read, so the paid verify turn runs), and a change drafted before the upgrade that still lists the isolation phase reads it by the restored groups; a change whose only target already complies ends not_needed (audit change.not_needed, metric outcome not_needed; rows that landed failed before stay failed); the change-window rule no longer folds a scanner finding into another incident, or an alert into a finding's incident; tickets that join a live incident are counted as joined, not suppressed (webhook response and ingest.batch audit); a kubectl cluster's estate ids and built-in detection alerts carry the connection's name — the next harvest removes the nodes written as k8s:cluster/…, and an alarm under such an id keeps clearing; EKS service-linked-role writes and kubectl auth whoami are no longer recorded (counted as service) and a pipeline's refused write is kept as a denied attempt; Test on a private FortiGate or PAN-OS management address answers inside_network; a FortiGate connection gains the read-only <name>_traffic_log tool. The changes run in the control plane; roll private-network agents as usual: an older agent does not offer certified step runbooks to a finding's analysis. The re-run setup output listing the four managed policies (Add) and the write role (Modify), or the deploy/aws plan's four aws_iam_policy.write and their attachments; Check the write connection's permissions on the certified patch runbook reading every permission is granted; agent version
0.1.156 Schema, additive and created at start: the column runnertoken.serves_chat; at that start every agent token that had already connected (the provisioner's aside) is marked as serving chat, so no private chat moves. New routes: POST /api/aiops/{tenant_id}/itsm-webhook-token, POST /api/aiops/{tenant_id}/k8s-audit-token/{cluster}, POST /users/{user_id}/reset-link, POST /api/aiops/{tenant_id}/changes/stats and POST /api/aiops/{tenant_id}/benchmarks/model-report. New settings: AIOPS_METRICS_TOKEN (control plane; empty = no check, the chart sets it) and AIOPS_KUBECTL_IN_CLUSTER (off; set only by the private agent chart's inClusterRbac mode). No switch default changes. Before the chart upgrade, metrics: the metrics listener answers 401 without Authorization: Bearer <AIOPS_METRICS_TOKEN> as soon as the control plane restarts, so first give every scrape job of port 9464 an authorization block reading the AIOPS_METRICS_TOKEN key of the release Secret (an annotation-driven job too: annotations carry no token), then upgrade; with existingSecret the listener stays open until that Secret holds AIOPS_METRICS_TOKEN: make Prometheus send the token first, then add the key, then set metrics.serviceMonitor.bearerToken: true if you use the ServiceMonitor. After the upgrade, change integrations (two breaking changes): (1) ServiceNow and Jira change tickets: for every workspace with aiops_unauthorized_change_incidents on, mint the ITSM webhook token — POST /api/aiops/{tenant_id}/itsm-webhook-token with members.manage (an admin-scope personal access token, or MCP mint_itsm_webhook_token; the token is shown once, minting again invalidates the previous one) — and re-point the ServiceNow business rule or Jira automation that forwards change tickets to send it as Authorization: Bearer <token> on /webhook/servicenow/<workspace-slug> or /webhook/jira/<workspace-slug> (alerts from the same webhook keep working with either token). Until then a ticket sent with the shared webhook token no longer accounts for a captured write, so the writes it covered are flagged as unauthorized; a ticket recorded earlier counts again on its next re-send with the token, and tickets sent through your own change feed never count. (2) Self-managed clusters' audit webhooks: for each cluster, mint its token — POST /api/aiops/{tenant_id}/k8s-audit-token/<cluster-name> with pillars.manage (an admin-scope personal access token, or MCP mint_k8s_audit_token; the portal has no control for it), <cluster-name> being the last path segment the cluster already posts to — put it in the audit webhook kubeconfig's token:, keep the server: URL ending in /-/<cluster-name> and restart the kube-apiserver. Until then the API server's deliveries with the webhook token are refused (401), the form without the cluster segment (…/kubernetes-audit/<slug> with or without ?cluster=) answers 404, and the writes they carry are not recorded; the API server drops what it cannot deliver. EKS clusters read through k8s_audit: true on the AWS connection need nothing (Kubernetes audit webhook token). Connections: a kubectl connection whose kubeconfig runs a credential plugin (exec or auth-provider: an EKS, GKE or AKS default kubeconfig) or names a file on the platform host (tokenFile, client-certificate, client-key, certificate-authority as a path) answers This kubeconfig runs a credential plugin … or names a file on the platform host … from every tool call, Test and inventory read until it is saved again with a static or service-account token and embedded data (certificate-authority-data, client-*-data; kubectl config view --raw --minify --flatten embeds them); a kubeconfig that resolves to no server ({}, no current context) now fails instead of reading the platform's own cluster. A Google Cloud connection saved without a service-account key fails closed: add the key (PATCH /api/aiops/{tenant_id}/datasources/{ds_id} with secret_values) or disable it. A keyless AWS connection saved with a ${field} placeholder in a key field fails closed until it is saved again, which binds the platform-minted External ID. Changing a write connection to another role, or the config of a write connection that holds a stored secret, now needs approvals.decide as well as datasources.manage (PATCH /api/aiops/{tenant_id}/datasources/{ds_id}): the builtin operator is refused 403; an owner, approver or executive is not. The ssh connector's -- lands on existing rows at the next start. Accounts and tokens: platform staff can no longer set a password on an account that holds a workspace membership or an organization seat, or that was born from single sign-on (409); use POST /users/{user_id}/reset-link (platform administrator, admin-scope token; needs a mail relay, the link goes to the account only) or let the workspace owner use Reset password on the member's row (POST /api/aiops/{tenant_id}/invites/{user_id}/reissue; emailed when a relay exists, else shown once). An account created by single sign-on that is not platform staff no longer signs in with a password: tell such users before the upgrade. A password change, reset or AIOPS_ADMIN_FORCE_PASSWORD now ends the account's other sessions. A personal access token now needs the admin scope for member seat and role changes, PUT …/idp, group-admin workspace staffing and the three token mints: re-mint a write-scope token a script uses for those. A private agent that should answer workspace chat is created by an owner with Answers workspace chat (members.manage); tokens that had connected before keep serving chat; staff-minted and operator-minted tokens never do. With AIOPS_SEED_DEMO off, the published demo webhook token is refused on every workspace: a workspace that still sends with it mints its own (POST /api/aiops/{tenant_id}/webhook-token). API callers: a script that saved a report with GET …/changes/stats?export=true or GET …/benchmarks/model-report?export=true now gets the read-only report; call POST at the same path with a write-scope token. A custom role that reads GET …/compliance/discovery-findings needs datasources.view too (every builtin role has it). aiops_compliance_muted in PATCH /api/aiops/{tenant_id} answers 400. Behaviour that changes on upgrade: a write an audit log captured is judged against the freeze window at the provider's audit time and a pushed or ticketed change at its receipt, so a write made in a freeze and pulled after it is flagged where it was not before; an L3 in-run approval counts a vote only for the command it released and only while the voter may still release the tier, so a pending L3 release with a stale or mismatched vote asks again; approval.* audit payloads written from now on carry the invocation with credentials hidden (older rows are shown hidden, the export is unchanged); a platform change no longer pins or folds alerts on its impacted-services list; a post-change round derived from a CloudWatch alert reads the alarm at the source, and an alarm CloudWatch does not confirm only flags a round; a rollback the provider refuses before any write releases its reservation and Roll back is offered again (a change locked that way before the upgrade stays locked); the platform's own Lambda runtime write is recorded under the function ARN; a deactivated workspace's (and its lab box's) failed changes wait for Activate instead of being rolled back automatically; a response from a workspace-chosen endpoint past 2 MiB or past the deadline is an error (a ticket system's or a git host's included: the ticket step fails, a write that got it is uncertain), and a lakehouse pull past 16 MiB an error status. The changes run in the control plane and in the agents: roll private-network agents as usual — an older agent still runs kubeconfig credential plugins, reads uncompressed answers without a cap and sends a compiled step re-rendered; its approval receipts are filtered on the control plane whatever its version. The ITSM webhook's next change ticket recorded with itsm_verified (GET /api/aiops/{tenant_id}/changes/{id}); a 200 on each cluster's next audit batch and its writes on the Changes page; curl of /metrics answering 401 without the token and aiops_build_info with it; a kubectl connection's Test passing after the re-save; agent version
0.1.157 Schema, additive and created at start: the column cloudfinding.cleared_reason, with a one-shot backfill (migration:cleared-out-of-scope-v1: a Backup or Patch Manager record the platform's read cleared in the 15 minutes before its out-of-scope audit row is marked out_of_scope and no longer counted as fixed). New routes: GET /api/aiops/{tenant_id}/runbook-templates, GET /api/aiops/{tenant_id}/incidents/summary, GET /api/aiops/{tenant_id}/changes/summary, POST /api/aiops/{tenant_id}/changes/{change_id}/schedule and POST /api/aiops/{tenant_id}/changes/{change_id}/schedule/cancel; no new permission. New workspace switches with their platform defaults: aiops_demote_on_deviation (on), aiops_canary_execution (off), aiops_repeated_other_resource (off); the settings map patch_schedules; the notification event change_schedule. No chart value changes. Nothing to do for the upgrade itself. Optional: re-run the guided AWS setup only for a shared account that wants the writer without the three account-wide statements (Leave out account-wide settings (shared account); the choice is not remembered on a re-run, so pick it each time). Behaviour that changes on upgrade: a certified step runbook whose backup or change target is a literal resource, or whose selector names one exact resource, no longer expands or classes a change (the route list and Needs you show the finding; give it a parameter bound from a label or annotation and certify it again); with aiops_demote_on_deviation on, a run through a certified runbook route that fails its validation or whose fix does not hold takes the route's certification away until an approver certifies it again (PATCH /api/aiops/{tenant_id}/guardrails demote_on_deviation: false keeps the 0.1.156 behaviour, where only a rollback demotes); every executed write step of a multi-step plan is followed by a read-only read of the resources it wrote (at most 10 a step, 40 a change) through the step's own connection, each a snapshot artifact — more read calls on the write role; the analysis prompt grows by the verdict-criteria line of the change candidates block (2,500 to 2,900 characters) and, when certified step runbooks match the incident, by a block of up to three runbooks, at most 900 characters each; a compliance or vulnerability change can be scheduled, and the scheduler releases it as the person who scheduled it with their role at that time (a removed or re-roled person's schedule lands refused, open to schedule again); a package-upgrade step needs ssm:SendCommand on AWS-RunShellScript on the write role (the guided role does not grant it: add it before certifying such a runbook). The demo workspace gains five disabled, uncertified step runbooks captured from Automated Security Response on AWS. Private-network agents need no change: the prompt additions reach an api-mode agent through the run's context from the control plane, and execution stays on the control plane. GET /healthz reporting 0.1.157; GET /api/aiops/{tenant_id}/runbook-templates answering seven templates; Check runbook reporting pinned_target on a runbook with a literal target; the Changes list's dashboard counts; the first change.scheduled audit row
0.1.164 No schema change, setting, route, permission or chart value change; new audit actions lkg_record.outside_explained and change.class_restored. Nothing to do for the upgrade. Behaviour that changes on upgrade: a runbook demoted for a recurrence whose outside write is harvested later is certified again (route.deviation_explained, restored: true) and that incident's unrequested change requests go back to standard; a demotion from before 0.1.164 waits for an approver. Private-network agents need no change. GET /healthz reporting 0.1.164
0.1.163 Schema, additive and created at start: the column cloudfinding.failing_since (no backfill: a finding already failing names no change until its next failing episode), and the one-shot migration:finding-family-v1 (classifies every stored finding again, 500 rows a batch, and writes the family only where it moved: DynamoDB.1, DynamoDB.6, ELB.6, Cognito.6 and RDS.8 rows leave the Backup list). New setting AIOPS_REQUIRE_BEFORE_STATE, the platform default of aiops_require_before_state (on). New audit actions incident.change_linked, change.finding_linked, compliance.finding_gone and compliance.scanner_out_of_scope. API additions: GET /api/aiops/{tenant_id}/alerts takes q and service (at most 200 characters each, else 422); GET /findings rows carry failing_since, the change detail upgrade_warning, and the harvest report and the manual pull's answer findings_linked. No new route, permission or chart value. Two platform defaults turn on: aiops_require_before_state and aiops_hold_for_approval, for every workspace that never set them. With the before-state rule on, a direct write runs only after its before-state read: a typed step over a CLI connection and a legacy direct route (api, cli or mcp) that declares no snapshot read become manual-only, and an approved or scheduled change on such a route is refused when it fires (the reviewed action changed: Manual route only…); pull requests and rollbacks are exempt. With the hold on, an escalated incident whose change request was requested (drafted, pending approval, or approved but not run), or whose proposal matched a certified runbook, no longer closes on the quiet window or when its ticket closes at the source; the first sweep audits incident.autoclose_held once per held incident. To keep the 0.1.162 behaviour, PATCH /api/aiops/{tenant_id}/guardrails {"hold_for_approval": false, "require_before_state": false}. A workspace that stored a value keeps it (on 0.1.162 an unticked before-state box stored false and read the same as unset): to follow the new default, send {"hold_for_approval": null, "require_before_state": null} and check that switch_state shows override: null for both. Behaviour that changes on upgrade: the next scanner pulls clear open Security Hub findings the hub no longer answers (last seen over 48 hours ago, every region read in full; a finding of another account than the connection's own only when the same pull returned a finding of that account, else once last seen over 90 days ago) and, while every pulling AWS connection has a tag scope, findings on harvested resources outside it; their incidents resolve, with the resolve notification and ticket write-back. A tag-scoped connection's Inspector read lands the CVEs on tagged repositories and functions, so a P1 or P2 one may start an analysis (aiops_auto_investigate_p1p2). R31 also fires between back-offs; a scoped workspace's denied attempts outside its scope open no SecOps report; a built-in metric alert joins its resource's service tag; a release whose quorum lapsed waits for approval again instead of being rejected; a finding_tag_scope without one key=value pair is refused at save (400). Thresholds and why is its own chapter, and old settings-reference#thresholds-* links land on its pointer section. The changes run in the control plane; a private-network agent on an image before 0.1.163 may leave out the change agent's upgrade warning, which then reads unknown, not checked. GET /healthz reporting 0.1.163; the control-plane log line online schema: finding-family-v1 refiled N finding(s); switch_state on GET /guardrails listing require_before_state; after an outside write and its scanner finding, the write's id in the incident's change_refs
0.1.162 No schema change, setting, route, permission, audit action or chart value change. Nothing to do for the upgrade. Behaviour that changes on upgrade: the findings list fits its card from 1280 px, the runbook catalog at 1024 and 1280 px and Who changed what on Changes at 1024 px (a long resource name or actor is cut, with the whole name on hover); a blocked route's operations on Problems break only after a dot; a change request whose blast radius was stored before 0.1.161 reads its specific reason alone, its folded coverage gap as Coverage gap recorded before 0.1.161, not split by kind: … and a bare No impact as the change management agent's note — on the change page and in the change request document, without rewriting the stored report; an RCA written after the upgrade heads its blast radius section with the change page's basis label (Blast radius — Estimated impact) instead of Blast radius (certified), and one written before keeps its text; the dark theme draws native controls dark. The changes run in the control plane: private-network agents need no change. GET /healthz reporting 0.1.162; the findings list, the runbook catalog and Who changed what fitting their card at 1280 and 1024 px without a sideways scroll
0.1.161 No schema change, no new route or permission. One new setting, aiops_dashboard_pillars (dashboard_pillars on GET/PATCH /api/aiops/{tid}/guardrails, members.manage; default incident, compliance, change); GET /api/aiops/{tid}/dashboard/summary carries shown_pillars, shown per ring and samples per mean. Nothing else to do for the upgrade. After it, the next estate harvest of each AWS read connection — the scheduled one (aiops_inventory_interval_s, daily by default), or Harvest now on the Estate map (every eligible connection) or on the connection in Data sources (datasources.manage; POST /api/aiops/{tid}/xray/harvest) — folds the duplicate Lambda function entries a map harvested before 0.1.161 holds (one $LATEST node per function) into the function's node; a function the map no longer holds keeps one node under its own name. A change request whose blast radius was stored before that harvest keeps it until its review or preview is opened again. Behaviour that changes on upgrade: the Dashboard shows three pillars — Incident & Problem, Compliance, Change Mgmt — until a workspace lists more under Guardrails → Dashboard pillars; the hub is one scalable picture; a ring and its Pillar health row read the same unresolved count (the column reads Unresolved); MTTA and MTTR print a comparison only from five or more incidents a side; list tables fit their card from 1024 px; the blast radius card and the change request document's 2.4 and 2.5 print one basis sentence and the map's coverage gaps by kind; the harvest keys a Lambda function's versions and aliases and a Step Functions state-machine version by the unqualified ARN (a Lambda layer version keeps its own node), so that first harvest also adds the versions and aliases properties to every function it reads and reports each such function once as changed — in the connection's harvest delta and as changed externally on the next Estate x-ray scan's Estate changes since the last scan card and the Estate map's Changes since the last scan mark; nothing changed in the account. The changes run in the control plane: private-network agents need no change. GET /healthz reporting 0.1.161; the Dashboard showing three rings around the human gate at any window width, the Unresolved column equal to each ring's number; the Dashboard pillars card at the end of Guardrails; after the harvest, no Estate map object named $LATEST
0.1.160 Schema, additive: the partial index ix_changerecord_tenant_cloud_write_executed (changerecord (tenant_id, executed_at) over harvested cloud writes — the read every did-not-hold mark now makes inside alert intake), registered at start and built in the background while the app serves, like every registry index (Schema changes after the pod is ready); on a large change ledger the build takes longer, and until the index is valid that read is slower, not wrong — nothing to do first. One new route, GET /api/aiops/{tid}/dashboard/summary (incidents.view; the spend figures need report.view; MCP dashboard_summary); the incident list and summary accept ?status=unresolved and the summary carries resolved_by_day; the Problems page reads the list's existing ?status= filter; a recorded fix gains the state recurred_after_outside_change and its recurrence marks the field outside_change_id. No new setting, permission, audit action or chart value. Nothing else to do for the upgrade. Behaviour that changes on upgrade: /portal/ opens the new Dashboard at /portal/dashboard (the former dashboard is replaced; a workspace with no enabled read connection still sees the Quick start once per browser session); a did-not-hold mark recorded after the upgrade names the recorded outside change that explains it and is not counted towards the ×N block, the candidate revocation or the not-good state — marks from before carry no such field and keep counting, existing problem rows are not rewritten. The changes run in the control plane: private-network agents need no change. GET /healthz reporting 0.1.160; the sidebar's Dashboard opening /portal/dashboard with the six rings and a number that opens its list with the same total; the index valid in the migrations block of GET /api/aiops/admin/overview (or indisvalid = true on the database); a Problems row reading recurred after an outside change ×N beside its automation facts
0.1.159 No schema change, setting, route, permission, audit action or chart value change; one new field, change_class_cause, on a change's meta and on an incident's proposed change. Nothing to do for the upgrade itself. Behaviour that changes on upgrade: a Lambda runtime-upgrade change (aws.lambda.update_runtime, aws.lambda.update_alias) is a typed fix: its validation plan's rows and its post-fix recheck read the function's configuration and alias back on the step's own connection and land the verdict at $0 once its derived metric window closes, with no agent run — a change still verifying at the upgrade is graded that way on its next check, and a verify round reading not read: no compiled check for a Lambda function is no longer expected; a released or verified Lambda fix stamps only the finding of the function it wrote to. A change whose standard class dropped before the upgrade carries no cause and reads runbook no longer backs it; one that drops after it says why (meta.change_class_cause). The changes run in the control plane: private-network agents need no change. GET /healthz reporting 0.1.159; a Lambda runtime-upgrade change's validation plan reading the function's configuration and alias as rows the platform read at $0; the runbook catalog's AWS-published rows reading Published by AWS · …
0.1.158 No schema change, setting, route, permission or chart value change; two new audit actions, route.deviation_explained and route.asr_seeded. Nothing to do for the upgrade itself. Behaviour that changes on upgrade: within one scheduler tick (a minute) of the start, every workspace with an enabled AWS write or read/write connection receives the five AWS-published step runbooks, disabled and not certified, bound to its oldest such connection (audit route.asr_seeded; a workspace that already holds one of the names keeps that route as it is and gets only the missing ones; a runbook a person deletes or renames afterwards is not added again) — a runbook catalog that lists Not certified · disabled rows it did not have before is expected, and nothing enables them; with aiops_demote_on_deviation on, a certified runbook route whose fix did not hold keeps its certification when the platform's own cloud audit harvest recorded a non-platform write on the same resource between the fix and the recurrence (route.deviation_explained on the recurrence incident; a run that fails its validation is still demoted), and a demotion recorded on 0.1.157 is not reversed — an approver certifies the route again; a scheduled release refused because the change class dropped lands with the full reason (what dropped, the approvals no longer count, review and release or schedule again). The changes run in the control plane: private-network agents need no change. GET /healthz reporting 0.1.158; the route.asr_seeded audit row on each workspace with an AWS write connection and its five runbooks in the runbook catalog; the Approvals card of an automatically released change reading Released automatically — Platform (pre-approved)

A version string from /healthz proves the control plane only. An older private-network agent ignores newer forwarded settings, so after every upgrade confirm the deployed agent version and one scoped datasource read.

Webhook body cap

From 0.1.149 the control plane answers 413 to a /webhook/* body over 1 MiB before it reads the body. Alertmanager, Grafana and the generic intake (/webhook/alertmanager/*, /webhook/grafana/*, /webhook/generic/*), which send a whole alert group in one notification, accept up to 8 MiB, and /webhook/kubernetes-audit/* up to 16 MiB. Earlier releases accepted any size where the load balancer in front sets no limit. A sender does not retry a 413: Alertmanager and Grafana drop that notification, and every resend of the same group fails the same way. An alert is roughly 1 KB from Alertmanager and 2 to 3 KB from Grafana, so a grouped notification of a few thousand alerts can reach the cap. Each 413 is logged as webhook '<source>': body ... bytes is over the ...-byte cap (the source path segment only, never the workspace or token) and counted in aiops_http_requests_total.

Where a source sends larger bodies, raise the cap through extraEnv (bytes; 0 removes the cap; the 8 MiB and 16 MiB sources keep the larger of theirs and this one):

extraEnv:
  - name: AIOPS_WEBHOOK_MAX_BODY_BYTES
    value: "16777216"

On the Alertmanager side, max_alerts on a webhook_configs entry keeps a notification under the cap by sending only the first that many alerts of a group; the rest are counted in truncatedAlerts and not delivered.

Kubernetes audit webhook token

From 0.1.156 /webhook/kubernetes-audit/<workspace-slug>/<token>/<cluster-name> takes only that cluster's own audit token, and the form without the cluster segment (/webhook/kubernetes-audit/<workspace-slug>[?cluster=]) answers 404. Earlier releases accepted the workspace webhook token there, which every alert and change integration holds, and recorded what it posted as the platform's own audit trail. After the upgrade the API server's deliveries with the webhook token are refused (401) and the writes they carry are not recorded. For every self-managed cluster that posts its audit log:

  1. Mint the cluster's token: POST /api/aiops/{workspace}/k8s-audit-token/<cluster-name> with pillars.manage (an admin-scope personal access token, or MCP mint_k8s_audit_token). The name is the last path segment the cluster already posts to. The token is shown once. The portal has no control for it: plan this step for someone who can call the API or MCP.
  2. Put it in the audit webhook kubeconfig's token: and keep the server: URL ending in /-/<cluster-name>.
  3. Restart the kube-apiserver and confirm a 200 on the next batch: the cluster's writes appear on the Changes page again.

EKS clusters read through k8s_audit: true on the AWS connection need nothing.

Schema changes after the pod is ready

From 0.1.149, before it serves, the control plane only adds tables and columns: a new column carries its default in the catalog, so no row is rewritten, and every statement waits at most 3 seconds for a table lock (or meets a deadlock), then rolls back and retries, up to 5 tries; the fifth waits in line for up to 60 seconds, so queries on that one table can pause for that minute once. The schema step therefore takes about as long on a year of history as on an empty database. Startup as a whole does not yet: with leader election off, the control plane also recovers the runs a restart interrupted before it serves, and that grows with the number of runs that were running. The chart's startup probe allows 5 minutes (60 checks, 5 seconds apart) before liveness counts.

Indexes on tables that already hold rows, and one-off data fixes, run afterwards, in the background, while the app serves:

To check one after an upgrade, on the database:

-- every registry index present and valid (false = a build that was interrupted)
SELECT c.relname, i.indisvalid
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname IN ('ix_incident_tenant_opened', 'ix_auditevent_action_target_at');
-- a build in progress, and how far it is
SELECT relid::regclass, phase, blocks_done, blocks_total FROM pg_stat_progress_create_index;
-- the data fixes: status running | done | failed, with the cursor
SELECT key, value FROM platformsetting WHERE key LIKE 'migration:%';

The full list is in Database schema. The control-plane log says online schema: building <index> and online schema: built <index> for each build, and online schema: migration:<id> done for each data fix.

An index left invalid (indisvalid = false: the pod was stopped mid-build, or the build was cancelled) is harmless: queries ignore it, and the builder drops and rebuilds it, after a minute when it failed while the pod runs (the wait doubles after each failure, up to an hour), at the next start otherwise. A data fix with failed is retried from its cursor the same way; its error field says why. The builder lifts statement_timeout and lock_timeout for its own connection, so a timeout set on the database role does not cancel a build.

To run the index builds yourself instead (a database whose changes a DBA makes by hand), set AIOPS_DB_ONLINE_INDEX_BUILDS=false through extraEnv. The control plane then logs, for every index still missing or invalid, the exact statement to run. Run them with a direct connection to the database: CREATE INDEX CONCURRENTLY cannot run inside a transaction, so it fails through a pooler in transaction mode.

Schema work that finishes after the upgrade

Some schema work runs in the background once the control plane is ready, so it does not hold up the rollout: indexes built without locking writes, and one-shot data migrations. GET /api/aiops/admin/overview (platform superadmin) reports it in its migrations block:

curl -s -H "Authorization: Bearer $SUPERADMIN_PAT" \
  https://aiops.example.com/api/aiops/admin/overview | jq .migrations

The upgrade is complete when pending is 0. The metrics port exports the same as aiops_migration_pending{name}: 1 until that index or migration is done, then 0. Keep the migrations block with the release record.

Verify the release

The following seven checks follow Verify a release in /ops/docs, Platform configuration. Checks 2–6 require a platform superadmin.

Check Pass condition
1. GET /healthz ok: true and the intended version: the image tag up to its first -, which is what the pipeline's own check compares; pipeline HTTPS/SNI certificate check passes
2. Ops → Customers and Workspaces Expected organizations and workspaces remain; their assignments are correct
3. Ops → Overview → Agent runners Expected execution mode and provisioner; active workspaces have live runners; no unexplained expired leases or exhausted retries
4. Ops → Connector catalog Expected connector types are present after startup sync
5. Ops → Platform defaults Saved model tier map survives; startup log confirms cache hydration if required
6. Ops → Audit log Fresh audit rows arrive
7. Portal smoke Incident opens, runner investigates and a human-reviewable remediation plan arrives

Retain deploy-log-<slug>: its deploy-log.json records image tag, digest, chart, snapshot, host, ALB and commit. The app health endpoint reports its version. Add the tenant smoke incident URL and outcome to the same release record. A model list or login alone does not prove runner execution.

The supplied smoke script, scripts/customer-smoke.sh, proves the runner part of check 7 from a host that reaches the platform, as in Install: an alert opens an incident and a runner investigates it on the real model. On an upgrade, always pass the workspace's current ingest token as WEBHOOK_TOKEN. With it unset the script mints a new webhook token, which replaces the workspace's token at once: every configured alert source then gets 401 until it is updated.

BASE=https://aiops.example.com TENANT=example-org \
  WEBHOOK_TOKEN="$(cat /secure/webhook-token.txt)" \
  ADMIN_PASSWORD_FILE=/secure/admin-password.txt scripts/customer-smoke.sh

The script signs in as ADMIN_EMAIL (default admin@local, the bootstrap superadmin), fires one generic alert for a new aiops-smoke-<epoch> service and waits up to TIMEOUT_S (default 1200 seconds) for the runner to finish the investigation. It fails unless a real model answered (per-model usage is recorded), the run booked a cost above zero and the RCA is longer than 50 characters; on success it prints SMOKE OK and the incident URL. It reads the investigation as that account, so the account must still be a member of the workspace: an HTTP 404 on that read means it is not. A TENANT slug that does not exist yet creates a new workspace, and workspaces cannot be deleted, so check the slug. The smoke costs a real model call, does not execute the proposed fix, and leaves its synthetic incident open for review.

Published documentation is part of the release gate. On every push to main the test pipeline (.azuredevops/azure-pipelines.yml, step docs leak scan) runs scripts/docs_leak_scan.py over everything the help site publishes — the product chapters, the repository docs outside the vendor-only folders and the film transcripts — and fails on any customer, person or host name. The forbidden-name list is the secret pipeline variable DOCS_LEAK_PATTERNS and exists nowhere else: not in the repository, not in the image. The scan logs only file, line and pattern index, never the matched text, and a run without the secret logs a visible warning instead of a pass.

Signature and immutable-tag checks

Signing is conditional. The release pipeline signs only when its signing key is configured; the deployment pipeline verifies only on that configured path. A warning about unsigned/unverified images is not successful signature verification. The published security review records signing as off; retain the actual pipeline result when that changes.

Registry image tags and OCI chart versions are immutable. The first build publishes the plain version tag and chart. For a rebuilt image of the same version use its full build tag; a chart change requires a new version. The chart currently pulls by tag even though the pipeline records the digest; digest pinning and cluster admission verification are not delivered.

Do not improvise a signing command or install a new trust key during an upgrade. Use the configured verification stage and retain its receipt. For a failed release follow Rollback, including the secret-store rotation boundary.

Sources: Deployment runbook §3, Availability, Platform configuration, Verify a release, and Security review AR-11, AR-12 and AR-19.

Next steps