Recurring checklist
Assign a named owner to each check below. These are proposed operating cadences for WBS 10.1–10.17, not a claim that every schedule or monthly service is already enabled.
Reference · Updated 2026-10-24 · Describes release 0.1.164
Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).
Assign a named owner to each check below. These are proposed operating cadences for WBS 10.1–10.17, not a claim that every schedule or monthly service is already enabled. Retain evidence for what actually ran.
Daily
Platform on-call checks external monitoring targets/rules, health and runner readiness, queued work, expired leases, intake errors and audit-seal backlog. Workspace owners check human escalations, approvals, failed verification and spend. QA checks yesterday's regression receipt if configured; missing receipt is not a pass.
Weekly
Review releases and security findings, capacity/expiry, failed runs and recurrent incidents. Review access changes and upcoming credential/licence expiry. Tuning owners inspect corrections, skill proposals and model quality before changing prompts or routing. Keep a small regression gate after each change; do not silently promote a draft skill.
Monthly WBS map
Each row also states its shorter cadence when relevant. The monthly report collects these results; it does not replace the daily response.
| WBS | Check and cadence | Mechanism / owner | Evidence and procedure |
|---|---|---|---|
| 10.1 | Platform watch and incident response — daily/on alert | External metrics, rules, health, runner leases and cancel/pause controls; platform on-call | Targets/rule health, delivery receipt, incident actions; Monitoring, Incident response |
| 10.2 | Safe operational changes — weekly/on change | Approved customer deployment, secret rotation, named release approver; release/cloud teams | Plan/apply record, deployment artifact, key-summary receipt; Upgrade, Security |
| 10.3 | Image/dependency patch review — weekly findings, monthly patch window | Enhanced registry scanning, pinned Python constraints and reviewed release; release/security teams | Running digest, complete findings with fixed versions, accepted exceptions and rescan; Patching |
| 10.4 | Model spend and guardrails — daily, monthly reconciliation | Priced models, run/day/month/incident limits, Pause all agents; workspace and platform owners | Limits, trip reasons, rate card and billing CSV; Cost |
| 10.5 | Operational performance — weekly, monthly report | Portal reports and incident outcomes; operations lead | Counts, duration/quality and explicitly defined denominators; report lacks a delivered auto-resolution-rate/SLA calculation, so calculate those separately and label the method |
| 10.6 | Service report and follow-up — monthly | Human operating review; operations lead | Monthly evidence record below; agreed actions and owners; Support |
| 10.7 | Access and audit integrity — weekly changes, monthly full review/archive | Users, Members & roles, IdP mappings, support grants, Verify chain and cursor Export; security/workspace owners | Signed access review, chain results and external archive manifest; Security, audit policy |
| 10.8 | Golden-set regression — scheduled daily if enabled, after changes | golden_check configuration, versioned dataset, same-mode comparison and verdict; QA/tuning owner |
Actual run id, dataset/model/fingerprint, pass rate and regression verdict; dataset seeding |
| 10.9 | Agent tuning — weekly/on repeated correction | Agent behaviour prompts and model evidence; tuning owner | Reason, before/after configuration and regression receipt; iteration playbook |
| 10.10 | Knowledge/skill curation — weekly | Human feedback → draft proposal → reviewed promotion; knowledge owner | Correction, proposal/reviewer and promoted skill version; Knowledge base |
| 10.11 | Model change and re-certification — on change | Model tier configuration plus benchmark compare/certify; platform/tuning owner | Exact model id, prices, comparable dataset/mode, old/new results and certificate; Agents and guardrails |
| 10.12 | Historical-incident back-test — monthly when usable history exists | Curated history dataset with accepted/corrected RCA and evidence replay; QA/customer incident owner | Provenance, validated-case selection, dataset version and results; no built-in multi-run certification trend report; dataset guide |
| 10.13 | Capacity and cost planning — monthly | Review workload, concurrency, estate sources and model/infrastructure invoice; cloud/finance and platform teams | Capacity headroom and actual priced invoices, assumptions separated from forecasts; Cost, Patching |
| 10.14 | Nightly lab subset — daily only when configured | Test Factory nightly settings, quotas and overlap guard on a disposable lab; QA owner | Actual subset/model/build, completed/failed/never-ran cases, budget and cleanup; Test Factory procedure |
| 10.15 | Cluster, node and certificate upkeep — weekly expiry/capacity, monthly maintenance | Cloud-team version review, Auto Mode nodes, ACM validation and renewal tracking | Versions/support horizon, node capacity, certificate expiry/renewal owner and change receipt; Patching |
| 10.16 | Backup and restore verification — daily backup visibility, monthly side-copy drill | RDS PITR and pre-upgrade snapshots; cloud/platform teams | Restore-point lag, available/verified timing, counts, audit replay, retained keys and copy cleanup; Backup and DR |
| 10.17 | AWS billing budgets/anomaly review — monthly/on alert | Customer payer-account Budgets and anomaly notifications; cloud-finance team | Invoice, forecast, budget thresholds and confirmed recipients; account ownership explicitly recorded; Cost |
Check that recurring automation is armed
- Golden runs require a valid
golden_checkcron anddataset_id; the scheduler does not create a default schedule. It avoids overlapping runs and has no catch-up burst. Inspect the configured dataset/model and actual run verdict, not merely the presence of scheduler code. - Nightly Test Factory requires
test_factory.nightly.enabled, UTC hour and subset configuration, working disposable estate, quota and budget. Review stored settings and last runs through the Test Factory API described in its procedure; a shipped feature does not prove it ran last night. - Pillar cron is UTC and best-effort through downtime. Saving refuses an expression the scheduler cannot read; a skipped maintenance window is not replayed afterwards.
- For released scanner scheduling, confirm the configured interval on Security posture → Posture → Scanner sync and read Last automatic pull. Off is a valid explicit choice; a saved interval does not prove a successful pull.
- For the 0.1.110 Posture trend, confirm yesterday is a solid saved day after the UTC close. Earlier reconstructed days are faded; today's outline is live. Record missing snapshot evidence for follow-up.
- Confirm real external alert recipients and SNS email subscriptions. The delivered monitoring receipt used an echo receiver; configure your on-call route and retain its own delivery test.
Reviews added between 0.1.111 and 0.1.130
- Pending risk requests: a request is not an approved exception until an approver other than the requester decides it (
aiops_risk_no_self_approvalrefuses self-approval where it is on). - Exceptions expiring or expired: read the recorded outcome. A displayed expiry date is not proof that the scanner cleared the finding.
- Attestations due for another human review: supersede, never edit, a past statement.
- Suppression at the source and its lift: review the receipt separately from the platform decision. A failed external write leaves the platform-side decision in force.
- The last retention sweep and the usage statements it sealed (Security operations).
- Routes awaiting certification on Needs you → Certify, and incidents whose timeline says Pre-approved route matched — not run: each is a decision an approver still owes.
A monthly record names the date, the installation and build, the owner, the scope, the actual result and the follow-up. A tick without its receipt does not establish a completed task.
Monthly operating evidence record
Keep this record with customer operations evidence, outside the product database where independent anchoring is required.
| Field | What to record |
|---|---|
| Period and reviewer | UTC start/end, installation/build, named operational and customer reviewers |
| Availability and incidents | Platform failures, impact, observed recovery time, open actions; estimates/SLA targets labelled separately |
| Spend | Booked operating, verify, discovery and test spend kept distinguishable; rates, CSV and provider/payer invoice reconciliation |
| Security | Access-review outcome, outstanding grants/tokens, patch findings and accepted exceptions with owners/dates |
| Audit | Full/window replay scope, sequence/head, archive files/hashes, predecessor anchor and write-once manifest |
| Recovery | Backup configuration, PITR lag, copy drill timing/counts/chain results, key custody and cleanup; cutover/decryption only if exercised |
| Quality | Dataset/case/model/build identity, actual completed and never-ran counts, comparable regression/certification results |
| Next actions | Decision, owner, due date and next verification; list missing evidence as missing |
Current open mechanisms include customer-side cutover/state-reconciliation proof, /data rebuild proof, automated access-review reporting, auto-resolution/SLA calculations, certification trend reporting, enforced audit truncation and scanner release gates. Track these as gaps rather than marking the recurring check complete because this runbook exists.
Sources: the chapter procedures linked per row, scheduler mechanisms, Test Factory reference, Availability, and Security review.