Changes reference

The change ledger record field by field — sources, statuses, classes, risk tiers, windows, last-known-good records, rollback paths, harvest sources and the audit actions a change emits.

Reference · Updated 2026-10-28 · Describes release 0.1.164

Exact facts about the change ledger. Procedures live in Change windows, emergency changes and rollback, Review execution in Changes, Runbooks and automatic fixes and Automatic API fixes; the approval rules are owned by Approval policy and change tiers.

The change record

One row per change the platform executed, prepared, declared or observed. GET /api/aiops/{workspace}/changes returns the public copy of each row (credentials masked structurally, the platform's own webhook token redacted, the internal execution_intent / rollback_intent keys removed from meta). On screen a change is quoted as CHG- plus the first 8 characters of its id.

meta keys the API exposes:

What the detail endpoint adds

GET /api/aiops/{workspace}/changes/{id} (incidents.view) returns:

Sources, origins and modes

mode on a platform change is the execution channel: gitops_pr, api, cli, mcp, network, or external (an operator reported it). On an ingested row it is the event kind:

List filters: kind=push,pr_merged,deploy,sync (the Deployments chip; an unknown kind is a 400), mode=platform (everything the platform executed, i.e. not cloud-write), source=, status=, service=, unauthorized=true (unauthorized changes, the Unauthorized chip; never a denied call), denied=true (calls the cloud refused — unauthorized access, the Denied attempts chip), requests=true (every request, proposals included, the ones waiting on a person first), include_proposed=true, q= (display id, title or service).

Statuses

Terminal states: completed, rolled_back, not_needed, rejected, recorded. A release is only possible from draft, planned, pending_approval or approved; an operator may report an external execution from those plus failed and verifying.

Change classes

The class is ITIL vocabulary decided in code at wrap-up, never by the model, and stamped as meta.change_class with change_class_source and a plain-words change_class_reason.

Derivation rules

Evaluated in this order; the first rule that applies decides. change_class_source names it.

A runbook route's standard class holds only while the runbook does. When the route is deleted, disabled, demoted, loses its certification or is edited before the request is released, the request reads as the class it has without the runbook (normal, or emergency for a P1 or critical incident) with source runbook_withdrawn on the incident and the change page — the class cell and the Timeline heading say why, in one vocabulary with the route row and the runbook catalog (meta.change_class_cause): certification withdrawn by a person, runbook demoted after a rollback, after a failed validation or after a fix that did not hold, runbook disabled, deleted, no longer certified or changed since the request was drafted; a row from before 0.1.159 carries no cause and reads runbook no longer backs it — at the tier that class raises it to (an emergency is L3, two approvers), the review releases it at that class, and the drop is recorded on the row (audit change.class_dropped, votes cleared) when a person next requests, approves or reclassifies it. An executed change keeps the class it ran under. A request is never stuck on its runbook: when the platform can no longer compile its steps without it — the runbook was deleted or disabled and its steps need it (an EC2 patch's isolation and patch steps run only inside a runbook), whatever the class — the release dialog shows Runbook no longer backs it — run it by hand with the runbook's steps in order: each check, the backup as its CLI command, each change as the command you run with your own access, each validate criterion and the rollback. Run them, then record what you did (on an incident, mark it resolved). A demoted or uncertified runbook still expands its steps: the request is a normal change the platform runs once a person releases it. A planned patch run whose runbook was deleted or disabled reads the same.

A route run from the incident page records standard with source runbook_route only when the run stays inside the route's declared scope (a GitOps route editing the file it declares, any other route running its own configured call); otherwise the row carries no class. The class only ever raises the release tier: risk_tier = max(intrinsic tier, class floor).

Standard by default

Guardrails offers two policies for change_class_policy:

The pre-approved operation list (aiops_preapproved_operations) is validated against the typed operation vocabulary (the cloud and Kubernetes operations plus the repository edit); whoever may edit routes cannot make a fix standard alone, because the owner's list on Guardrails must also name every operation.

Reclassifying

POST …/changes/{id}/class with {change_class, note} (incidents.act, note at least 5 characters):

Validation plan

Every change request an analysis drafts carries the list L6 checks, fixed at the draft (meta.validation_plan) and bound into the approval digest by the review (execution_spec.validation_plan) with the expected blast radius from the impact data (expected_blast_radius: the intended resources and the lost facts, never the affected list, so a harvest cannot stale an approval). At most 10 items, each {id, target, read, expected, group, direction, hops}:

The post-fix recheck runs exactly that list: each changed resource whose executed step compiled a read is read first, by code, on the step's own connection (no sooner than 30 s after the write) — a start or reboot by its power state, the groups an isolation recorded by every network interface listing them again (an isolation left in place: only the isolation group), a patch by its own command's status (ssm:GetCommandInvocation) and Patch Manager's missing and failed counts (ssm:DescribeInstancePatchStates), a Lambda runtime upgrade by its function's configuration (lambda:GetFunctionConfiguration: the runtime the step sent, LastUpdateStatus Successful, State Active or Inactive) and by its alias (lambda:GetAlias: the version the run published), and a read that cannot answer is not observed, never pass; the verify turn is told only the listed items, with no web research; a row it adds for anything else is dropped; every item ends with a row (item = its id), observed or not; and a listed resource the platform read as failing decides the verdict (still_present). The verdict rests on the listed rows only: a still_present the turn based on a row it was not asked for reads uncertain, and so does a cleared turn while a listed item — a neighbour included — reads fail. A failing neighbour never makes the change still_present on its own. Once every alert of the incident cleared at its source after the write, the platform reads the list without a verify turn — when the alert source's clear arrives, and on each scheduled check after it — and lands cleared at $0 when every listed changed resource reads pass and no neighbour probe fails; anything else leaves the verdict to the turn (switch verify_typed_check). A standard change runs it too. A change drafted before the validation plan existed carries no list and is verified as before.

Pre-change backups

With aiops_backup_before_change on (the default), the review puts a backup step before every reboot-type step on a stateful target — aws.ec2.reboot gets aws.ec2.create_image (an AMI with NoReboot=true), azure.vm.restart gets azure.compute.disk_snapshot (a snapshot of the VM's OS disk) — once per target, through that step's own write connection. aws.ec2.create_snapshot (an EBS volume) is the same kind of step for a runbook to declare. A step runbook's own backup step is compiled the same way at its place in the runbook, bound to the write connection of the change step after it, even with the switch off; a target it already backs up gets no second backup. Backup operations are compiled by code and never offered to the analysis. A backup needs the native API connection: over a CLI connection the review refuses the plan and says why.

The platform never deletes a backup, expired or not: removing old images and snapshots is the workspace's own lifecycle rule. The write role needs ec2:CreateImage, ec2:CreateSnapshot, ec2:CreateTags, ec2:CreateReplaceRootVolumeTask, ec2:DescribeImages, ec2:DescribeSnapshots, ec2:DescribeReplaceRootVolumeTasks on AWS and Microsoft.Compute/disks/read, Microsoft.Compute/snapshots/read, Microsoft.Compute/snapshots/write on Azure (re-run the setup to add them). The restore reads its image or snapshot only when it carries the aiops:change tag, a backup the platform made: an AMI that exists is authorized by its own tags, never by the request tag the restore puts on its task.

Risk tiers

The tier stamped on a change is tiers.classify over the execution channel, the incident's production labels and its trusted blast radius, raised by the route's tier_override and by the change class; the policy per tier (approvers, permission, on-screen definition) is the table in Approval policy and change tiers. What this page adds: where each tier is computed.

Change and freeze windows

Both windows share one shape and one matcher; they read opposite policies off the same answer.

The emergency bypass: a change whose release class is emergency passes a closed change window once its full quorum is collected (L3 = two distinct people); the first pass stamps meta.window_bypassed and audits change.window_bypassed once, so the re-checks at dispatch and at each step are idempotent. A route run's row is never waved through: it carries no class, or a standard nobody can reclassify.

Last-known-good records

Before a platform write the state it is about to change is captured and stored as an append-only artifact (kind: lkg); the change's lkg field says what was captured and how. The incident page's "Last-known-good record" card (Read an incident and its analysis) reads the same versions.

Capture methods

Fields on a capture: artifact_id, path, repo, sha, captured_at, prestate_status (present / absent / unknown), source_observed_at, source_version, completeness (complete / partial / unknown), health ({status: unverified} until a verify reads it), sha256, restorable: false when the stored evidence is not byte-exact, and evidence_artifact_id for a redacted receipt. Compliance control CHG-02 counts a change as covered only with a real method (gitops_file, gitops_create, route_snapshot, or any captures[] entry other than none).

Version sources

Every ledger row yields resource versions (lkg.versions_of); the source says where the before-state came from and the record card labels it.

A harvested AWS write without Config history reads "external AWS write — no before-state captured (AWS Config history off for this connection)"; one where Config answered but every capture it returned was after the write says so too.

The before-state rule

settings['aiops_require_before_state']: true — a write whose before-state cannot be captured never runs on the platform: the review refuses it up front (manual_only, with the operator-run steps), and a read that fails at execution stops before the write (audit change.prestate_blocked). On by default since 0.1.163 (null follows the platform default AIOPS_REQUIRE_BEFORE_STATE); false lets the write proceed with an honest none. A pull request is exempt: the repository's base commit is its before-state.

Rollback paths

rollback.kind and rollback.generated_by say what the platform holds. A rollback is never auto-executed unless the sweep's matrix below says execute; a human releases one with POST …/changes/{id}/rollback (approvals.decide; an L3 rollback needs a second distinct approver; the original route must still exist, be enabled and keep its mode).

An agent may draft a rollback for a manual change (the plan runner); the draft is shown, never executed. Every non-manual rollback also gets a kind: rollback artifact with the plan rendered for people (artifact_id_doc).

The automatic rollback sweep

Scheduler tick over platform changes with status failed updated in the last aiops_rollback_sweep_window_h hours (24), 50 per tick, each decided once (audit markers change.auto_rollback, change.rollback_escalated, change.rollback_skipped). The decision is pure:

Escalation pushes the rollback notification event. The sweep deliberately re-checks no change window. After a rollback of a run executed through a pre-approved route, settings['aiops_demote_on_rollback']: true takes that route's pre-approval away at the first rollback (Needs you: route_demoted; audit route.demoted).

Harvest sources

The scheduled harvest reads every eligible connection's write events since its last marker minus a 20-minute overlap (the ledger dedupes on the event id, and a re-seen event carrying a diff the first sighting lacked enriches its row). Interval: aiops_change_harvest_interval_s = 900 s platform default; settings['aiops_change_harvest_interval_s'] overrides it, 0 or less disables the scheduled harvest for the workspace. Per-connection opt-out: harvest_changes: false. A write-role connection is never harvested.

CI/CD service accounts on Kubernetes

A pipeline that applies manifests with its own service account releases a change the same way a person running kubectl does, so its writes are recorded with actor_kind: service and the namespace as ci_namespace. Which namespaces count is the workspace setting k8s_ci_namespaces; the platform default is flux-system, jenkins, gitlab-runner and tekton (a Tekton pipeline often runs as a service account in its own pipeline namespace: add that namespace). argocd cannot be listed, because Argo CD syncs arrive on /webhook/change/<slug> and would be recorded twice, and kube-system, kube-public and kube-node-lease never can. Every other system: identity (nodes, controllers, other service accounts) stays plumbing.

What a pipeline writes that is not its change is left out: the pods and runs it spawns (pods, taskruns, pipelineruns, customruns, runs), status writes, objects of its own controller (*.fluxcd.io, *.tekton.dev) and the per-job objects GitLab Runner names runner-<token>-project-<id>-concurrent-<n>… (build pods, registry secrets, services). A server-side dry run (dryRun=All — Flux compares every object it manages this way on every reconcile) is never a change, for a person or a pipeline. On EKS the query leaves all of these out, so they are not counted. Per pull, an object the same account wrote many times is one row (the newest write). At most 50 objects are recorded per cluster, service account and clock hour (UTC), counted after the ledger's dedupe — what an earlier, overlapping pull already recorded in that hour counts first, in time order — so overlapping pulls agree on which 50; the rest are counted as noise, and the pull that reaches the cap writes the marker note once. A denied attempt (401/403) is never left out as one of the objects above, folded or counted toward the cap: it is recorded as a denied attempt and reaches SecOps like a person's (a pipeline refused creating pods in kube-system included). The audit webhook applies the same rules and has no marker: what it folds goes to the platform log. People and pipelines share the 1,000 audit rows one pull reads per cluster.

A pipeline's change is checked against freezes and references like any other; to let it through without a ticket, add system:serviceaccount:<namespace>:* to change_authorized_principals.

Caps per connection per tick: 500 rows recorded (1,500 raw read), request parameters and diffs scrubbed then cut at 2,048 characters; values written to Parameter Store, Secrets Manager, AppConfig or a Lambda / ECS environment are stored as [not stored] (which key changed is kept). The manual pull (POST …/changes/harvest, pillars.manage, "last N minutes") ignores the interval and lands under its own marker so it never blinds the next scheduled tick.

Service resolution for an observed change: the event's own service → settings['change_service_map'] ({repo_glob: service}) → unmapped (recorded, listed under an empty service). settings['change_resource_map'] ({resource_glob: service | [services]}, at most 200 globs, 10 services each) only adds linked services; editing it relinks the rows of the last 48 hours; a cloud write to an EC2 instance, a volume, an SQS queue or an SNS topic also links itself to a service named after the resource.

Principal classes and unauthorized changes

For every harvested cloud write that is not the echo of a platform receipt, meta.principal says who made it:

A row is flagged authorized: false when one of these holds. The last two are unauthorized changes (audit change.unauthorized_detected, the Unauthorized chip); a denied call is unauthorized access, not a change (audit change.denied_attempt, the Denied attempts chip), and the change statistics count it apart: never in the change total, a source, a service or the unauthorized changes:

A platform write seen back in the audit log within 10 minutes becomes the receipt of its change (change.confirmed_by_audit_log), never a second row. Only the platform's own harvest may fold a write this way; a webhook sender cannot stamp "confirmed by the cloud audit log" on a change.

Unauthorized changes and denied access

The method per change source, in prose, with a sandbox demo: How unauthorized changes are detected.

Off by default. aiops_unauthorized_change_incidents (platform default AIOPS_UNAUTHORIZED_CHANGE_INCIDENTS, off; the workspace's own true/false on Guardrails → Unmanaged changes → Open an incident for every change with no reference) makes every applied write the audit log captures — AWS CloudTrail, the Azure activity log, the EKS audit log or the Kubernetes audit webhook — need a reference. A busy account records thousands of writes a day made outside the platform: fill change_authorized_principals with your CI/CD roles before turning it on. Denied attempts are never judged by this rule; the existing noise, service-identity, read and failed-event filters still apply first.

A write has a reference when, checked in this order:

With none, the row is flagged authorized: false with "no reference found (no ticket, approval or signed change request)" (audit change.unauthorized_detected).

ITSM ticket fields (settings['itsm_change_match'], each optional; a ticket's re-send refreshes the fields on its row, so a ticket approved after it was raised counts from then on):

Times without a zone are read as UTC.

The ITSM webhook token. The shared webhook token is held by every alert and change integration, so it cannot vouch for an approval. POST /api/aiops/{workspace}/itsm-webhook-token (members.manage, MCP mint_itsm_webhook_token; audit tenant.itsm_webhook_token) returns a separate token once and stores only its hash; minting again invalidates the previous one. A personal access token needs the admin scope for this call. Send it from the ServiceNow or Jira webhook (Authorization: Bearer <token> on /webhook/servicenow/<workspace-slug> or /webhook/jira/<workspace-slug>, which also accept it for their alerts). A ticket and its re-sends refresh the ticket fields only when they carry this token. Tickets recorded before the workspace had one are not references: the next re-send with the token makes them count.

What an unauthorized change opens:

The person's two answers, on the incident's Unauthorized change card (approvals.decide):

• Approve rollback opens the drafted revert's change page: review the exact action, approve and release it — validation and the incident's close follow the change as any other, the validation being the rollback's own read at no cost (see Verify recovery), and the incident's Timeline lists the rollback's request, approval and execution. A revert with no typed step has nothing the platform can run, so it shows Open the command instead: its change page holds the command; run it with your own access, then Mark resolved.
• Keep this change (legitimate) asks for a note (10 characters or more): every open revert is rejected, every change of the incident is marked authorized: true with "kept as legitimate: <note>" and meta.kept, and the incident closes as expected (noise, no verification). Audit change.kept per change and incident.change_kept on the incident. Refused (409) once a rollback ran or runs.

Denied attempts (unauthorized access): with aiops_denied_access_to_secops on (platform default AIOPS_DENIED_ACCESS_TO_SECOPS, on) and the SecOps agent enabled, every denied attempt opens or joins one SecOps incident per principal, account and clock hour — one alert row per attempt (source denied-access), correlation key denied_access=<account>/<principal>@<hour>, P3, no analysis started, no rollback (nothing changed). Its Denied access attempts card lists the refused calls.

On a workspace with a tag scope (the workspace scope, or a connection's finding_tag_scope), an AWS or Azure attempt opens a report only when it was made as the identity of one of the workspace's connections, or its target is an estate resource inside the scope: one account shared by several teams, another team's role refused on another team's resource is theirs. The rest stay on the ledger as denied attempts and are counted on the harvest's report (denied_out_of_scope). Kubernetes audit attempts come from the workspace's own cluster and always count.

What is never captured

The incident card, change_lookup and the last-change-check skill say this, so an empty list reads as "nothing in what we can see":

• settings changed inside an application (its admin screen or its own database), unless the application posts them to the change webhook
• Kubernetes objects edited with kubectl on a cluster without the audit webhook or EKS audit log — only pushes to watched repositories and Argo CD syncs are seen
• files, registry and packages inside a VM or container
• data-plane writes: S3 objects, DynamoDB items, Key Vault secrets or App Configuration key-values written through their own endpoint (their ARM writes are captured)
• the values written to Parameter Store, Secrets Manager, AppConfig or a Lambda or ECS environment (which key changed is kept)
• AWS writes in regions a connection does not declare (only the declared regions plus us-east-1 are read)
• cloud control-plane writes at all, while no AWS or Azure connection is harvesting

Post-implementation review

Audit actions

Every row targets the change id (the incident Timeline and /audit show the same rows with the labels below).

Settings

Workspace settings that change how the ledger behaves (Guardrails unless noted). Platform defaults are the control plane's environment.

API summary

Next steps

• Next: Compliance reference
• All Reference pages
• Documentation home