Post-upgrade checks, 0.1.144 to 0.1.149

The operator follow-up and the evidence to keep for every release from 0.1.144 to 0.1.149, for an install older than 0.1.150: the second part of the table in Upgrade and release verification, moved…

Reference · Updated 2026-10-29 · Describes release 0.1.173

The operator follow-up and the evidence to keep for every release from 0.1.144 to 0.1.149, for an install older than 0.1.150: the second part of the table in Upgrade and release verification, moved here when that page reached its size ceiling again. An install older than 0.1.144 reads Post-upgrade checks, 0.1.109 to 0.1.143 first. Read every release between the installed version and the target, oldest first, then continue with 0.1.150 and later on that page. The release notes are the authority; three versions are never installed (0.1.106, 0.1.115 and 0.1.127 — see that page).

Release crossed Operator follow-up Evidence to retain
0.1.144 Schema, additive and created at start: new table compliancereport (one row per workspace per UTC day) and the JSON column investigation.triage. Re-run the AWS CloudShell setup for every account it set up (a role from the deploy/aws module: re-apply it). The read role gains config:DescribeConformancePacks, config:DescribeConformancePackCompliance, ssm:DescribeInstanceInformation, ssm:DescribeInstancePatchStates, backup:ListBackupJobs, ec2:DescribeVpcEndpoints, ec2:DescribeTransitGateways, ec2:DescribeTransitGatewayAttachments, ec2:DescribeTransitGatewayVpcAttachments, ec2:DescribeTransitGatewayRouteTables and ec2:SearchTransitGatewayRoutes (a role on ReadOnlyAccess already has them). The write role gains ec2:CreateImage, ec2:CreateSnapshot and ec2:CreateReplaceRootVolumeTask on instances and volumes tagged aiops-managed=true, the same three plus ec2:CreateTags on images, snapshots and restore tasks created with the aiops:change tag, and ec2:DescribeImages, ec2:DescribeSnapshots and ec2:DescribeReplaceRootVolumeTasks; nothing that deletes. Re-run the Azure Cloud Shell setup for a write connection: its role gains Microsoft.Compute/disks/read, Microsoft.Compute/snapshots/read and Microsoft.Compute/snapshots/write. The guided read role also gains health:DescribeEvents for the automated triage's cloud-health step (a role on ReadOnlyAccess already has it); until the setup is re-run, that step reads not available. Until the setups are re-run, Scores lists the new reads as denied and a reboot stops at its backup step (UnauthorizedOperation) before anything changes. Defaults that change behaviour on upgrade: a backup before every reboot-type step (AIOPS_BACKUP_BEFORE_CHANGE, on; kept 7 days by tag, never deleted by the platform; a reboot-type plan over a CLI write connection is refused at review); a certified step runbook carries its own operations (AIOPS_RUNBOOK_OWNS_OPERATIONS, on): Pre-approved operations covers single-call routes only, a fetched skill alone no longer makes a change standard, and pre-approving a fix waits until its incident is resolved; a person may only raise a change class, and a P1 a certified runbook covers is released under the standard rules (requests recorded before keep their stamped rules); denied access attempts open SecOps incidents while the SecOps agent is enabled (AIOPS_DENIED_ACCESS_TO_SECOPS, on); a scanner HIGH maps to error (P2) instead of critical (P1) (register rows change on the scanner's next send, open incidents keep their priority); a P3/P4 finding with a prepared fix waits for Analyse; the automated triage reads cloud health and known issues before each incident analysis and adds ## Fault domain to its report (AIOPS_TRIAGE_CHECKLIST, on); the compliance reports are read once a day (AIOPS_COMPLIANCE_REPORT_INTERVAL_S = 86400). Off until a workspace turns them on: unauthorized-change incidents (AIOPS_UNAUTHORIZED_CHANGE_INCIDENTS; fill change_authorized_principals first), several runbooks per change (AIOPS_RUNBOOK_MULTI_COVER), the change management agent (AIOPS_CHANGE_AGENT_CR) and its fix drafting (AIOPS_CHANGE_AGENT_OWNS_PLAN); AIOPS_PATCH_WITHIN_DAYS is 0 (no rule). A private-network agent in database mode needs this release for the triage steps on its runs (in API mode the control plane does them); turning on the change management agent's fix drafting needs every private-network agent on this release. The re-run setup output; the first Scores report with no denied call; GET /guardrails switch_state per workspace; one incident's Automated triage card
0.1.145 No schema change. Re-run the AWS CloudShell setup for every write connection it set up: the write role gains iam:SimulatePrincipalPolicy on its own ARN only, with an explicit deny on every other principal (a role from the deploy/aws module is unchanged and its permission check keeps reading unknown). Until the check can answer, the step-runbook automatic lane does not run a runbook on its own; a person releases it. Behaviour that changes on upgrade: an unconfirmed permission check stops an automatic step-runbook run; a change request the platform executed stops covering outside writes 60 seconds after its execution; STS calls are no longer harvested as changes; a fix whose alerts cleared at the source and whose validation plan reads as expected clears with no agent run; the agreed comparison table's sha256 changes (A.8.8 and A.8.13 notes), so a recorded acceptance reads as changed since and needs a new acceptance; a reviewed revert that re-creates a security-group rule with a description needs a new review (its digest now carries the description). The re-run setup output; a step runbook's Check runbook showing the permission check passed; GET /guardrails switch_state per workspace
0.1.146 No schema change. Re-run the AWS CloudShell setup for every account it set up (a role from the deploy/aws module: re-apply it). The read role gains lambda:ListFunctions, ecr:DescribeRepositories, ecr:DescribeImages and ecr:ListTagsForResource (a role on ReadOnlyAccess already has them); until the setup is re-run, the harvest's Coverage lists lambda:ListFunctions as denied. The write role gains ssm:SendCommand with the AWS-RunPatchBaseline document only on instances tagged aiops-managed=true, ssm:GetCommandInvocation, ssm:ListCommandInvocations and ssm:DescribeInstanceInformation, and Reachability Analyzer (ec2:CreateNetworkInsightsPath and ec2:StartNetworkInsightsAnalysis only with the aiops:change tag on create, ec2:StartNetworkInsightsAnalysis on a path and ec2:DeleteNetworkInsightsPath and ec2:DeleteNetworkInsightsAnalysis only on what carries it, account-wide ec2:Describe* (instance and launch-template user data included), ec2:GetManagedPrefixListEntries, ec2:GetTransitGatewayRouteTablePropagations, ec2:SearchTransitGatewayRoutes, elasticloadbalancing:Describe*, network-firewall:Describe* / List* and tiros:CreateQuery, ExtendQuery, GetQueryAnswer, GetQueryExplanation, GetQueryExtensionAccounts); nothing uses those yet. ssm:GetCommandInvocation and ListCommandInvocations are on * and return any command's output in the region, and AWS-RunPatchBaseline takes a BaselineOverride and an InstallOverrideList IAM cannot restrict, so the write role can install packages on a tagged instance. The backup's ec2:CreateTags is now bound to the backup's own create calls (ec2:CreateAction = CreateImage, CreateSnapshot, CreateReplaceRootVolumeTask) and to the tag keys aiops:change and aiops:retain-until. A deploy/aws write role also gains iam:SimulatePrincipalPolicy on its own ARN and a region pin on its aiops:change-scoped statements. Defaults that change behaviour on upgrade: the scheduled Security Hub / Defender pull is on every 15 minutes (AIOPS_SCANNER_SYNC_INTERVAL_S = 900; a workspace that stored 0 stays off), whatever the SOC 2 / ISO 27001 switch says, and its Inspector CVE read is on unless a connection says pull_vulnerabilities: false. With Analyse P1/P2 findings automatically on, every HIGH or CRITICAL CVE starts one paid analysis (measured USD 1.04 to 2.92 each): before the first pull, set each workspace's daily_cost_limit_usd, set Scanner sync to off where the pull is not wanted, and give an AWS read connection into a shared account Only findings on resources tagged (finding_tag_scope, one key=value; any other value fails that connection with invalid_tag_scope, it never pulls wider). The change management agent drafts every change request (AIOPS_CHANGE_AGENT_CR, on): one change-agent run of at most 8 steps per change request, about USD 0.33 measured. A workspace that set aiops_change_agent_owns_plan: true without setting aiops_change_agent_cr now gets Phase 2 (the agent drafts the plan); set aiops_change_agent_cr or aiops_change_agent_owns_plan explicitly before the upgrade if that is not wanted. Screens and names: the change page's "Verification" card is Validation plan and "Safety net" is Rollback plan; "Alert correlation" is Service correlation in the sidebar and on L2 (skill id alert-correlation, the /alerts route and the unmapped filter are unchanged); the first firewall harvest lists deny rules as added once. API and MCP: the automated triage's triage_checklist[*].step numbers follow the new order (1 dependency map, 2 known issues, 3 cloud health, 4 recent changes, 5 signals, 6 summary, 7 fault domain); match a step by its title, not its number. The re-run setup output; the next harvest's Coverage with no denied lambda:ListFunctions; a step runbook's Check runbook permission check; GET /findings/sync (interval_s, interval_inherited) and GET /guardrails switch_state per workspace
0.1.147 Schema, additive and created at start: new table syntheticcheck, the columns incident.merged_into_id and investigation.scope_deferred (JSON), and the index ix_incident_tenant_opened on incident (tenant_id, opened_at). No new permission. New routes: POST /api/aiops/{tenant_id}/changes/{change_id}/reject (approvals.decide), GET and POST /api/aiops/{tenant_id}/monitors, PATCH and DELETE …/monitors/{check_id}, POST …/monitors/{check_id}/run and POST …/monitors/derive (datasources.view / datasources.manage), with the MCP tools reject_change, list_monitors, create_monitor, update_monitor, delete_monitor, run_monitor and derive_monitors. The storm gate is removed: aiops_storm_threshold and aiops_storm_window_s (a workspace setting, or AIOPS_STORM_THRESHOLD / AIOPS_STORM_WINDOW_S on the control plane) are accepted and ignored, and a burst of unrelated services now opens one incident each (the shared-alert-name fold, aiops_fold_shared_cause_threshold 3, stays). Grouping reads the new windows aiops_storm_group_window_s (300 s, at most 3600) and aiops_storm_spread_window_s (600 s, at most 7200), whole seconds, platform AIOPS_STORM_GROUP_WINDOW_S / AIOPS_STORM_SPREAD_WINDOW_S or a workspace setting; aiops_storm_agent keeps its meaning (the grouping agent, on). A __storm__ incident left open is not split any more: no alert joins it, it resolves when its alerts clear or a person resolves it, and an alert on one of its services opens that service's own incident beside it. A dashboard or alert on aiops_webhooks_total{outcome="storm"} must read outcome="grouped". SOC 2 / ISO 27001 are off in every workspace (AIOPS_COMPLIANCE_FRAMEWORKS, off): the first scheduler tick after the upgrade resolves every open platform-check finding (control:<id>, by: scope-change; one a person holds stays), and a compliance scan answers 409. AIOPS_COMPLIANCE_FRAMEWORKS is platform-wide: true (helm extraEnv) turns SOC 2 / ISO 27001, EST-01 to EST-06 and the platform alarms back on in every workspace whose compliance.enabled is not an explicit false, customer workspaces included. Set it only once the customer has agreed the comparison table, never for a Test Lab twin on a shared control plane; Test Lab cases that expect the platform's own checks get no scan while it is off. Defaults that change behaviour on upgrade: the known-issue fast path (AIOPS_KNOWN_ISSUE_FAST_PATH, on: an alert-started analysis rated 100 % relevant runs capped at 3 steps); post-change validation in 3 rounds 300 s apart (a rule's minutes no longer times anything: validation_rounds: 1 with validation_interval_s = minutes × 60 reads about the old single window); validation signals derived from the change request's plan when no rule matches (AIOPS_VALIDATION_DERIVED, on); a 10-minute hold on the automatic verify after a start, reboot or restart; daily synthetic-check suggestions (AIOPS_MONITOR_DERIVE, on): up to 20 internet-facing load balancers and API Gateway APIs per workspace are probed from the control plane, GET / every 300 s, in trial (nothing opens) until a person keeps or discards them; an API Gateway call is billed and can invoke its integration, so turn aiops_monitor_derive off where that is unwanted; backup findings from the daily AWS Backup read (AIOPS_BACKUP_FINDINGS, on: one P3 incident per workspace while SOC 2 / ISO 27001 are off); CI/CD service-account writes recorded as changes (AIOPS_K8S_CI_NAMESPACES: flux-system, jenkins, gitlab-runner, tekton); Azure DevOps workitem.updated deliveries that change the state are signals, so a service hook that already sends them now resolves incidents on Done, Closed, Resolved or Removed, and a state change on a work item the platform has not seen opens an incident with an analysis (any other update is ignored; an incoming ticket missing required fields is not held yet); AWS Config findings key on their own rule (control:<rule-name>): a finding already open in the shared incident stays there until it clears, new ones open one incident per rule. Roll private-network agents to this release: an agent in database mode needs it for the service's resources in the prompt, a run's resource selection and the vulnerable-package plan per resource type. GET /guardrails per workspace: switch_state.known_issue_fast_path, verification.validation_derived and compliance_frameworks_available: false; the platform-check findings resolved by: scope-change; the first incident.grouped audit row; Synthetic checks after the first daily suggestion; agent version
0.1.148 No schema change or permission. New workspace setting aiops_grouping_wait_s (whole seconds 0–600, default 300, 0 = no grouping wait; PATCH /api/aiops/{tenant_id}), separate from aiops_storm_group_window_s: a workspace that raised the group window keeps a 300 s wait. New audit action change.approval_lapsed. New route POST /api/aiops/{tenant_id}/changes/{change_id}/close-without-verdict (approvals.decide, body {"reason": "…"}; 409 until the change has gone 24 hours without a verdict), with the MCP tool close_change_without_verdict. An approved change released by a person who did not vote runs again; a change the 0.1.147 dispatch re-check set to rejected stays rejected, so its incident needs a new change request. Behaviour that changes on upgrade: an incident on a service that depends, over a dependency allowed to fold, on another service holds its analysis for the grouping wait (aiops_grouping_wait_s, 300 s) while its ticket and notification go out as usual, and an alert that clears inside the wait resolves its incident with no analysis (when it comes back it is analysed at once); with a reopen window set, an incident a person closed as known noise is no longer reopened (the next alert on its key opens a new incident, analysed as usual), and a caller whose resolved incident would reopen joins its root's open incident instead. One-shot clean-up of the connection tag scope (Only findings on resources tagged): AWS Backup records already open for a resource outside it clear at the next daily read of their region (one compliance.backup_out_of_scope audit row; a tag none of the resources read carries clears nothing and the Backup card says so); suggested synthetic checks nobody kept whose resource is outside it are removed at the next daily suggestion run, also where suggestions are turned off (named in the monitor.derived row); built-in detection pairs already in alarm on such a resource resolve (outside the workspace scope or the connection's tag scope), except under a scope that cannot be read or that none of the connection's resources carries, which resolves nothing. Security Hub findings of DynamoDB.1, the deletion-protection controls, ElastiCache.1/.2, FSx.1/.2 and Redshift.3 move family when Security Hub next sends them (daily). The rollback sweep no longer escalates a change that stopped at its backup. The fixes run in the control plane; roll private-network agents as usual. An approved change released by a second approver reaching change.executed; a dispatch refusal's change.release_refused row naming its reason; the first compliance.backup_out_of_scope row and the Backup card's count; a caller incident reading Held for triage — grouping window: …; agent version
0.1.149 Image and chart bump, no values change; no new table or column. The start's schema step no longer scans tables: each statement waits at most 3 s for a table lock and retries (5 tries, the last waiting up to 60 s); with leader election off the start still resumes the runs the restart interrupted before it serves. Once the app serves, the control plane builds seven new indexes with CREATE INDEX CONCURRENTLY (ix_auditevent_action_target_at, ix_auditevent_tenant_target_at, ix_incident_tenant_key_status, ix_incident_tenant_status_resolved, ix_investigation_tenant_started, ix_knowledgeitem_tenant_name_version, ix_changerecord_live_status; minutes on a large audit trail, writes continue) and runs migration:null-defaults-v1 once (Schema changes after the pod is ready); both need a direct database connection, not a pooler in transaction mode. A workspace without a PII salt gets one at the first start. The chart adds a startup probe (60 × 5 s) before liveness counts. A webhook body over 1 MiB is answered 413 (Alertmanager, Grafana and the generic intake: 8 MiB; kubernetes-audit: 16 MiB) and the sender does not retry it: where a source posts larger groups, raise AIOPS_WEBHOOK_MAX_BODY_BYTES through extraEnv or set Alertmanager's max_alerts (Webhook body cap). aiops_change_records exports open statuses only: a dashboard or alert on completed, rolled_back, rejected or recorded reads nothing. New series: aiops_scheduler_subtick_seconds, aiops_migration_pending. Rollback: the previous image with this chart kept (Rollback); nothing in the database is undone and the new indexes stay. The migrations block of GET /api/aiops/admin/overview with pending: 0 (every index valid, null-defaults-v1 done); /healthz version; the pod's time from start to Ready (under 30 s when no run was interrupted); every 413 line in the control-plane log, with its source

Next steps