# Change windows, emergency changes and rollback — set it up like the demo

> Set change and freeze windows, release an emergency change with its review, verify a fix and roll a change back.

_How-to · Updated 2026-10-28 · Describes release 0.1.164_

## Goal

Review an exact change, enforce its release rules, verify its actual result and restore recorded state when appropriate. A proposal, an approval, a write receipt and a successful verification are distinct events.

## Prerequisites

- A correctly scoped change or incident, writer consent and supported operation.
- `members.manage` for workspace window/safety settings; `incidents.act` to request work; `approvals.decide` to release the required quorum.
- A known before-state/recovery method, maintenance coordination and outcome evidence.

## Procedure

1. Open the change request and read class, class reason, tier, scope, approval count, current state and the blast-radius basis. **Derived** and **Standard by default** are policy choices; **Reclassify** raises the class (normal to emergency, never lower), records a note and clears prior votes; a certified runbook's standard moves only with its runbook. A “runbook matched” skill input is different from a pre-approved runbook route or the current revision-bound route certification. **You should see** the actual recorded reason and quorum for this request.

   API: `GET /api/aiops/{tid}/changes/{id}` — `incidents.view`; `POST /api/aiops/{tid}/changes/{id}/class` — `incidents.act`, with the new class and explanation.

![A change request: the Change request document card with its Markdown and Word downloads, then class Standard — runbook route with its reason, tier L2 · live infra · 1 approver, scope Whole plan, the L5.2 approval with reviewer and review digest, the window that applies, and the Plan → Approval → Apply + snapshot → Verify → Complete line.](/help/media/portal/portal-change-standard.png?v=0.1.146)

2. Configure a release window deliberately. The **Maintenance windows** card edits a workspace default plus service glob rows. The first matching service row wins; a service with no match falls back to the workspace row, and no declared row leaves releases open. Typed plans use the resolved service window; routes keep their own `params.change_window`. The change and incident pages show **Window that applies:** `<service> → <window>`. Times are UTC and overnight hours wrap midnight; the hours after midnight belong to the day the window starts, so the example below is open from Saturday 22:00 to Sunday 06:00 (not Saturday 00:00–06:00, which is Friday's night). **Weeks of the month (1–5, blank = every week)** picks which of those weekdays: `{"days":["tue"],"weeks":[2],"hours":"22:00-02:00"}` is the 2nd Tuesday from 22:00 to Wednesday 02:00, read as *2nd tue 22:00-02:00 UTC*; a month without a 5th such weekday never opens week 5. The same rules apply to a route's own window, the automatic lane and `change_freeze`. **You should see** a read-back of the exact saved body and a preview's actual open/closed verdict before relying on the restriction.

   API: `PATCH /api/aiops/{tid}/guardrails` — `members.manage`, for example:

   ```json
   {"change_window":{"days":["sat"],"hours":"22:00-06:00"},"change_windows":{"payments*":{"days":["sun"],"hours":"22:00-02:00"}}}
   ```

   A malformed/empty release window can fail open on the checked implementation. Validate the shape and inspect the preview rather than assuming a typo restricts execution. Setting `change_window` to `{}` clears the legacy workspace alias too; no window is then declared.

   To patch a service in its window, add a row to the **Patch schedules** card under the windows: **Service** (the same pattern as its window row), **Plan ahead (hours)** (the lead time, 1–336, default 24), **Owner (email)** and optionally the **Patch runbook id** (blank = the first enabled step runbook with a patch step). A schedule is refused while no maintenance window applies to its service. At its lead time before the window opens, the platform plans one change request, *Patch run: `<service>` — N instances*, for the EC2 instances of that service Patch Manager reports non-compliant (the open Patching findings, and Security Hub's SSM.2): an instance belongs to the service when its `service`, `app`, `application` or `Name` tag, or its instance id, matches the pattern (our proposal). The request lists the instances in its scope and compiles the patch runbook once per instance — backup, isolation, patch, restore, every target bound by code; with no patch runbook it carries the commands to run by hand. It waits for approval, and the owner and the notification targets that take `change_schedule` are told. **Review execution** opens with the next maintenance window chosen: approve it and it runs when the window opens, as scheduled (step 4) — never on its own. One run per window opening, at most 25 instances (the rest wait for the next window). A run nobody approved before its window opened is rejected (*the maintenance window opened before anyone approved this patch run*) — one still waiting for a second approver when the window opens counts as unapproved — and the next window plans a new one (our proposal). Audit: `change.patch_planned`. API: `PATCH /api/aiops/{tid}/guardrails` — `members.manage`, `{"patch_schedules": {"payments*": {"lead_hours": 24, "owner": "app-owner@example.com"}}}`; `null` or `{}` clears.

   If a window holds one particular change, select **Request a one-time window** on that change. Enter a UTC start, end no more than 24 hours apart, and a mandatory reason. The requester needs `incidents.act`; a person with `approvals.decide` selects **Approve one-time window** or **Reject**. The requester cannot approve while another approver exists; sole-approver self-approval is audited. Approval is bound to this exact action digest, opens only this change during the interval, and is not a release vote. The digest and audit trail state who approved it and when; changing the reviewed plan or window makes the exception unusable and requires a new decision.

3. Distinguish a **freeze window** from a release window. `change_freeze` flags observed outside changes within that window as unauthorized; it is not a universal prevention control for all external systems. **You should see** the observed event, matching time and recorded authorization reason.

   API: workspace PATCH with `settings.change_freeze` uses the same UTC day/hour shape — `members.manage`.

![Changes: Pull the last 15 minutes, and What the change agent watches — the cloud audit logs it pulls and the repositories each git provider polls, with a failed poll and a provider that watches no repository flagged in red; Who changed what starts below.](/help/media/portal/portal-changes.png?v=0.1.132)

4. Review the action and release only through the current exact digest. Normal changes are at least L2; emergency changes are L3 with two distinct approvals. A standard PR can remain L1 because the PR is inert until merged, while a standard live-infrastructure change remains subject to the relevant L2/window rules. A Standard record for a P1/critical incident can retain emergency release rules, including two distinct approvers; read the recorded class reason and release policy. **You should see** the actual preview gate and the recorded first vote separately from execution.

   API: change `/execution-preview`, `/approve` and `/execute`; release requires `approvals.decide` and the tier/window checks.

   A change that writes to several resources can run one resource first: with **Run a multi-resource change one resource first** on (Guardrails → Pre-approved operations, off by default), the review says *Canary: … runs first* and the release runs the first resource's steps, reads it back with each step's own check, then the rest. A failed check stops the run before any other resource is touched and the change fails with only that resource written, so only that one needs rolling back — released as any failed change's rollback, automatic only where the workspace allows it; a check that cannot observe it does not stop the run. The first resource is the one the plan writes first (our proposal); a plan that interleaves resources runs as before. The change page records the read under **What ran**.

   A compliance or vulnerability change can run later instead. In **Review execution** its dialog shows **When to run**: **Now**, **At a time (UTC)**, or the next opening of its service's maintenance window (the window and the time it opens are shown); the button then reads **Approve and schedule**. Your release vote on the exact reviewed action is recorded at once. At that time the platform releases it as you through the same checks as a release now: the action unchanged (same digest), the window and the approval quorum checked again, and your role then still allowed to release it. At L2 and L3 a picked time outside the window is refused (`422`); a GitOps change takes any time, since it only opens a pull request that stays inert until a person merges it. Until then the change reads *Scheduled to run …* with **Reschedule** and **Cancel schedule** (cancelling keeps the approvals), and the Changes list shows *Scheduled ·* its time. If the release is refused at its time (the action changed, the window, an approver missing, your role), the change reads *Not run at its scheduled time* with the reason and stays approved for a person; when the reviewed action changed because the change class dropped after you scheduled it (its certified runbook was demoted or disabled), the reason says so — *The change class dropped from standard to normal after this release was scheduled (…); the approvals it had no longer count. Review the change again, then release it or schedule it again.* — on the change, in the audit row and in the notification; a release that ran but did not complete reads *Released at its scheduled time. It did not complete:* the reason, and the change itself is failed. A release the platform loses mid-way (a restart) reads *Releasing* and is reported as not run within the hour. An incident's fix is never scheduled: it runs as soon as it is released. Each step is audited (`change.scheduled`, `change.rescheduled`, `change.schedule_cancelled`, `change.schedule_released`, `change.schedule_refused`) and sent to the notification targets that take `change_schedule`. Offering both a picked time and the maintenance window is our proposal while that choice is open with your change process.

   API: `POST /api/aiops/{tid}/changes/{id}/schedule` — `incidents.act` or `approvals.decide` (the tier decides who may release), body `{"review_digest": "…", "at": "2026-11-01T02:00:00Z"}` or `{"review_digest": "…", "window": true}`; `422` for an incident's change, a past time, more than 62 days ahead, no window, or a time outside the window at L2/L3; `409` when the action changed since the review. `POST /api/aiops/{tid}/changes/{id}/schedule/cancel` — same permissions; `409` when nothing is scheduled. A new schedule on a pending one reschedules it.

   To refuse a request instead, select **Reject** beside the approval buttons and give a reason. The request becomes **rejected**: it leaves the waiting queue, its approval ledger shows **Rejected by** with the person and time, the incident's remediation step reads *Not released —* your reason, and the incident itself stays open for a person to re-analyse or close. That plan is not offered again: requesting or releasing it answers *This plan was rejected (…) — re-analyse for a new plan*, and a re-analysis drafts a new one. Only a request nobody released can be rejected (draft, planned, waiting for approval or approved); anything dispatched or finished is refused. Closing or merging the incident rejects its unreleased requests the same way. API: `POST /api/aiops/{tid}/changes/{id}/reject` — `approvals.decide`, body `{"reason": "…"}` (1–500 characters).

   A released change whose verification never concludes does not wait forever. Once it has gone 24 hours without a verdict — none at all, or only **uncertain** ones (counted from the first uncertain verdict after the write, or from the write when none was recorded) — a person with `approvals.decide` sees **Close without verdict** on its page and gives a reason. The change becomes **failed** with the verdict **uncertain** and *closed without a verdict —* your reason as its note, and the audit trail gets a `change.closed_without_verdict` row. Nothing rolls back on the close: the rollback sweep waits while the verdict stays uncertain (a change an operator reported, or one in a computer-assist workspace, is handed to a person as usual) and **Roll back** stays yours to release. A later verdict still decides: cleared completes it, still present goes to the rollback sweep like any failed fix. API: `POST /api/aiops/{tid}/changes/{id}/close-without-verdict` — `approvals.decide`, body `{"reason": "…"}` (1–500 characters); `409` before the 24 hours or for any other status or verdict.

![A change request parked at its quorum, below its Change request document card: pending approval, L3, 1 of 2 approvals recorded; the release still depends on the change window.](/help/media/portal/portal-change-request.png?v=0.1.146)

5. For an emergency, the approval that completes the quorum can permit the closed-window bypass. Inspect the bypass audit. The PIR is scheduled at submission by default since 0.1.133 (off: it opens when the write lands). **You should see** the review's opened/due time and pending/implemented outcome; the first vote is not a write receipt.

   API: guardrails PATCH with `pir_at_submission` — `members.manage`; the review closes via `POST /api/aiops/{tid}/changes/{id}/pir/close` — `approvals.decide`, with a substantive outcome note. A pending implementation cannot be presented as completed.

6. Distinguish **Change safety → Post-change validation** rules from the separate **Validation checks (L6)** card. If a metric rule matches the executed change, inspect the actual rule, threshold and period, and the rounds (three, five minutes apart by default; a rule's `minutes` no longer times anything); the rounds decide before the compiled typed-fix check. Each round is read three minutes after it closes, and the change page lists every round with what it read. Every round failed fails the fix; every round clean with the alerts cleared at their source validates it; anything else goes to the verify. A gap or missing data is not equivalent to every period failing. Automatic compiled cloud rollback is limited by the configured L1/L2 policy and other guards; L3 still needs people. **You should see** the validation verdict, rollback decision and resource read-back rather than only a rollback plan.

   API: guardrails PATCH with `validation_metrics`, `validation_rounds`, `validation_interval_s`, `auto_rollback_cloud`, `reopen_raise_severity` and `audit_correlation` — `members.manage`. Enable correlation before the lifecycle starts; it does not retroactively stamp older rows.

   Without a matching metric window, **Check a typed cloud fix with its own read first** can verify supported operations through their compiled target read about 30 seconds after the write, and clears any other fix once every alert of its incident cleared at the source after the write and the changed resources on its validation plan read as expected. **Skip agent checks that cannot learn anything new** avoids redundant paid checks; both options default on. **Treat an alarm still firing at its source as still present** defaults off and changes verification behavior. Each follows the platform default until overridden. A verdict reached this way is badged **platform verdict** on the change page. **Verify now** always requests the agent, so read the automatic result before requesting additional paid work. **Spend** and the L6 line show the actual AI runs, failed runs and free checks.

   API: guardrails PATCH fields `verify_typed_check`, `verify_frugal` and `verify_alert_rung` — `members.manage`; `null` follows the platform default. A resource-verified finding marker is separate from the scanner’s later clear.

![Guardrails → Change safety: refuse a change without a before-state (on by default), raise severity on a failed fix, PIR at submission, back up before a reboot-type change, block pulls of a vulnerable image, correlation ids, backup retention and Roll back a failed cloud fix automatically.](/help/media/portal/portal-guardrails-change-safety.png?v=0.1.164)

![Guardrails → Validation checks (L6): Check a typed cloud fix with its own read first, Skip agent checks that cannot learn anything new, Treat an alarm still firing as still present — each beside its platform default.](/help/media/portal/portal-guardrails-validation.png?v=0.1.132)

![An executed change, Validation plan: the declared criteria and the dependents' advisory checks, then the cleared verdict badged with who decided it and the run that produced it.](/help/media/portal/portal-change-verification.png?v=0.1.146)

7. For a human rollback, use **Roll back** on the executed change or **Undo the executed change → Review and roll back** on the incident. Review the stored inverse and applicable quorum. Follow `rolling_back` to the actual outcome and **Verify restoration only**. **You should see** the rollback receipt and restored-state evidence.

   API: `POST /api/aiops/{tid}/changes/{id}/rollback` — `approvals.decide`; restoration verification uses `/rollback-verify` — `incidents.act`. If the outcome is uncertain, use **Reconcile rollback outcome** rather than retrying the write.

![A completed change: the Change request document card, then tier, scope and approvals, the pipeline line, Verify now and Roll back, the incident and pull-request links, the blast radius, and the plan reused from the investigation's wrap-up.](/help/media/portal/portal-change-completed.png?v=0.1.146)

8. Understand the automatic sweep's decision before expecting a restore. **You should see** a recorded action, wait, skip or escalation reason matching this case:

   | Situation | Expected policy boundary |
   |---|---|
   | Eligible GitOps fix failed validation | A restore PR may be opened; it remains inert until merged |
   | Provider refused before acting | Skip a restore of a write that never happened |
   | The plan stopped at its backup | Skip: no write step ran, nothing was applied (a person who reconciled the write as executed overrides this) |
   | Forward/rollback outcome uncertain | Wait for reconciliation |
   | Compiled cloud inverse | Human release unless the bounded failed-metric automatic policy applies |
   | Operator-declared rollback script | Requires its configured opt-in/tier policy |
   | External work, computer-assist or unsupported/manual recovery | Human escalation, not an invented executable inverse |

![An executed change, Rollback plan: the pre-change state captured before the write and the rollback by code with the exact reversed edit; the verification verdict the sweep reads sits on the Validation plan above.](/help/media/portal/portal-change-safety-net.png?v=0.1.146)

9. For work done outside the platform, use **Execution mode → Computer assist** and the documented **Record external execution**, **Record deployment evidence** or **Reconcile outcome** action. **You should see** an operator-reported receipt distinguished from a platform-executed write; merely recording it does not execute it.

   API: execution mode uses guardrails PATCH — `members.manage`; `POST /changes/{id}/external-executions` and `/application` (under `/api/aiops/{tid}`) require `incidents.act`; `/reconcile-execution` also requires `approvals.decide`.

![Guardrails → Execution mode: Standard or Computer assist for the workspace; under computer assist the platform makes no writes and operators record external execution.](/help/media/portal/portal-guardrails-execution-mode.png?v=0.1.132)

10. To propose a separate planned change, create it through the API/MCP, then **Draft plan with the change agent**. Drafting can spend model budget. **You should see** the resulting plan waiting for review — its **Plan** card reads *Drafted by the change agent*, as on the request pictured under step 4 — without a write merely because the plan exists.

    API: `POST /api/aiops/{tid}/changes` / MCP `create_change`, then `POST /api/aiops/{tid}/changes/{id}/plan`; verify the caller's current `incidents.act` permission and plan scope.

## What the ledger records as a change {#what-counts}

| Write | Recorded | How it arrives |
|---|---|---|
| A person's write to a cloud account | Yes | CloudTrail / Azure activity log harvest |
| A person's `kubectl` write (a pod delete included) | Yes | EKS audit log (`k8s_audit: true`) or the audit webhook |
| A CI/CD pipeline's service account in a namespace listed in `k8s_ci_namespaces` | Yes, one row per object per pull, at most 50 objects per account | The same audit log ([CI/CD service accounts](/help/changes-reference.md#k8s-ci)) |
| An Argo CD sync, a push or merge on a watched repository | Yes | Their webhooks and the repository harvest |
| The cluster's own controllers, kubelet, leases, events, other service accounts | No | Left out of the EKS query (not read, not counted); the audit webhook drops them |
| A server-side dry run (`dryRun=All`, how Flux compares every object it manages) | No | Left out the same way: nothing changed |
| Pods a controller restarts on its own, nodes the cloud replaces | Not today; still open | — |

## Verify and cleanup

Retain the exact request, independent approvals, before-state, provider/PR outcome, verification, rollback and PIR. Restore temporary windows/switches with read-back proof. Keep planned work and evidence intact after a refused or uncertain action. Never repeat a write to tidy the record.

## Troubleshooting

| Symptom | Check | Next step |
|---|---|---|
| Change window closed | UTC body, class/tier, actual preview verdict | Use the agreed window or the genuine emergency quorum; do not relabel to bypass policy |
| Manual route only | Supported operation, writer state, consent and before-read | Repair the named prerequisite and review afresh |
| Failed with provider refusal | Provider text and whether any action occurred | Preserve the failed receipt; do not assume a rollback is needed |
| Receipt pending | Actual resource state and request identity | Reconcile before any retry |
| Route demoted after rollback | Current certification and rollback record | An approver reviews and certifies the revision again, or keeps it manual; a tick alone is not proof of certification |
| PIR cannot close | Implementation outcome still pending | Record the real outcome first |

## Related

[Approved fixes](/help/setup-write-connections.md) · [Incidents](/help/incidents.md) · [Guardrails and cost limits](/help/guardrails-and-cost-limits.md) · [Report changes to the change webhook](/help/change-webhook.md) · [Audit log](/help/audit-log.md) · [Runbooks and automatic fixes](/help/runbooks-and-automatic-fixes.md) · [Automatic API fixes](/help/automatic-api-fixes.md).


## Next steps

- Next: [How unauthorized changes are detected](/help/unauthorized-changes.md)
- [All Work with the agents pages](/help/work-with-the-agents/llms.txt)
- [Documentation home](/help/)