Availability and recovery
As of 0.1.110 (8 October 2026): what a customer can rely on today, and what is roadmap.
Reference · Updated 2026-10-24 · Describes release 0.1.164
As of 0.1.110 (8 October 2026): what a customer can rely on today, and what is roadmap.
The control plane runs as one pod behind the ALB health check: when the pod crashes or is evicted, Kubernetes replaces it in roughly 30–60 s; when its node is lost, the replacement first waits for Kubernetes to give up on the node and release the /data volume — up to about 12 minutes, not yet measured. Every upgrade is a Recreate rollout with about a minute of downtime. The agents run in a separate runner pod per workspace and every job is durable: a runner that dies mid-investigation is re-queued when its lease expires (≤ 3 minutes) and the next runner resumes from the per-step committed transcript, up to 3 attempts. An api-mode runner (the customer and demo default) checkpoints every step through the control plane, so a step that checkpoints while the control plane is down — during an upgrade too — is lost the same way and the run waits out its lease. The database is RDS PostgreSQL, Multi-AZ in the customer template, with 7 days of automated backups (point-in-time restore) plus a manual snapshot before every upgrade. The control plane's /data volume lives in one Availability Zone, so losing that zone stops the control plane until the zone returns, whatever the database does. Leader election for a multi-pod control plane exists (app/aiops/leader.py, a Postgres advisory lock) and is on in the demo, but the platform ships one replica because the per-tenant lake sits on a single-node volume. A multi-replica, multi-AZ control plane is roadmap, not a setting.
Recovery objectives
| Failure | RPO | RTO | Notes |
|---|---|---|---|
| Control-plane pod crash or eviction | 0 for committed state; one step for an api-mode run that checkpoints in the gap | ~30–60 s; that run resumes as its next attempt once its 180 s lease expires | demo ALB: check every 5 s, 2 healthy, 10 s drain; an api-mode runner retries a refused checkpoint for about 3.5 s, then drops the attempt |
Upgrade (Recreate) |
as a pod crash | ~1 min, planned; the same lease wait for such a run | db-mode runners keep going: the DB rows are the queue. From 0.1.149 the start does not grow with the data: before serving, the pod only adds tables and columns (a default in the catalog, no row rewritten, each statement giving up a table lock after 3 s and retrying); indexes and data fixes run after it serves (CREATE INDEX CONCURRENTLY, one pod at a time) |
| Node loss | 0 for committed state | up to ≈ 12 min, not yet measured | pods leave an unreachable node only after Kubernetes' default 300 s not-ready/unreachable toleration (the chart sets no shorter one), then the ReadWriteOnce /data volume waits up to 6 min for a forced detach; a runner on that node is evicted on the same 300 s clock, so its jobs resume after about 6 min, not 3 |
| Runner pod mid-job | one step (per-step transcript commit) | ≤ 3 min lease, then resume | aiops_job_lease_s=180, aiops_job_max_attempts=3 — a run out of attempts fails and its incident is escalated; chat, benchmark and API-facade runs are not resumed |
| RDS instance or host | 0 (synchronous Multi-AZ standby) | ~1–2 min, automatic failover | single-AZ (rds_multi_az = false): no standby, so recovery takes longer, and lost storage means a restore, as below |
| Bad data or a bad release (restore) | up to about 10 min (PITR: RDS ships transaction logs about every 5 minutes), or the pre-upgrade snapshot; measured 8 min 14 s at the restore point on a customer-template install (samples from 3 min 03 s to 8 min 14 s) and 2 min 34 s to 4 min 12 s on the hosted sandbox | tens of minutes — a restore creates a NEW instance, which then takes over the original's name. Measured on the hosted sandbox (7 October 2026, 20 GB): 11 min to a restored copy, 14 min to a verified one (drill record); on a customer-template install (8 October 2026, 20 GB): 19 min to a restored copy, 22 min to a verified one (drill record); the cut-over is not drilled. The customer template grants the database login (rds-db:connect) to user aiops on any instance of its account (dbuser:*/aiops, iam unit), so the control plane logs in to the restored instance unchanged; the hosted sandbox, and an install rendered from an older template, pin the original instance's resource ID, which a restored instance does not share, so there the iam unit must be applied again first |
runbook §4 Rollback |
| Availability Zone | 0 for the database (Multi-AZ) | database ~1–2 min; the control plane is down until the zone returns when its /data volume is in that zone |
egress runs through one NAT gateway unless single_nat_gateway = false |
DuckDB lake on /data |
not backed up | re-collect from the sources | see below |
Measured on the demo cluster (3 October 2026, 0.1.104)
Two runner-kill drills on a throwaway workspace with a real model:
| Scenario | What happened | Time to completion |
|---|---|---|
Planned: kubectl delete pod on the runner (rollout, drain) |
the old process finished the run inside its 300 s drain while the replacement pod was already up; the attempt counter stayed 0 | 119 s after the delete |
Unplanned: hard kill (--grace-period=0 --force) at step 2 |
the replacement pod was up in 3 s, waited out the 180 s lease, then resumed from the committed transcript as attempt 1 (steps 2 → 8) | 301 s after the kill |
No run was lost or duplicated and the incident kept one history. Node loss has not been drilled; its row above is an estimate.
Database: the RDS settings
From infrastructure/iac/envs/_customer-template (rds/terragrunt.hcl, env.hcl).
| Setting | Value | Knob |
|---|---|---|
| Engine, class, storage type | PostgreSQL 16, db.t4g.medium (~340 connections), gp3 |
rds_instance_class |
| Storage | 20 GB, autoscaling up to 100 GB | rds_allocated_storage, rds_max_allocated_storage |
| Multi-AZ | on: a synchronous standby in the second AZ, about twice the instance cost | rds_multi_az |
| Automated backups | 7 days, point-in-time restore | rds_backup_retention_period |
| Pre-upgrade snapshot | aiops-<slug>-pre-<tag>-<buildId>, taken by deploy-customer.yml before every rollout |
— |
| Snapshot tags | the stack's tags are copied to every automated and manual snapshot | — |
| Deletion | deletion_protection, final snapshot on destroy |
— |
| Alarms | free storage below 2 GiB, connections above 80 % of the class limit; they notify the customer's SNS topics and, with alarm_email set, the topic aiops-<slug>-alarms that mails that address (a named recipient) |
alarm_sns_topic_arns, alarm_email |
| Egress | one NAT gateway in one AZ | single_nat_gateway (false = a second NAT gateway, AZ-resilient egress) |
| AWS service endpoints | an S3 gateway endpoint and ECR, Secrets Manager, STS and Bedrock interface endpoints with private DNS: cached image pulls, secret reads, role assumption and model calls stay on AWS's network; the NAT still carries the first pull through the cache, webhooks and tickets | enable_core_endpoints, enable_bedrock_endpoint |
The hosted sandbox database (aiops-demo-pg) differs from the template: single-AZ in ap-southeast-1a while the control plane's /data volume is in ap-southeast-1b (losing either zone stops it), db.t4g.medium, PostgreSQL 16 with automatic minor upgrades. It had no automated backups until 7 October 2026; since then it keeps 7 days with point-in-time restore, deletion protection and a final snapshot on destroy. Its two alarms exist but notify no one (alarm_email and alarm_sns_topic_arns are unset in the sandbox host environment's env.hcl). Its windows are the ones AWS assigned (the IaC sets none): backups daily 15:41–16:11 UTC, maintenance Tuesdays 14:08–14:38 UTC. An optional operating-system update (system-update) is pending and is not applied before the handover. Upgrades there roll through ArgoCD image-updater and take no pre-upgrade snapshot.
Authentication is IAM (signed 15-minute tokens); the master password lives in Secrets Manager and is read only by the first-install step, so no database password needs backing up.
What is NOT backed up
The control plane's /data volume (ReadWriteOnce EBS) has no volume snapshot, no Data Lifecycle Manager policy and no AWS Backup plan, by decision (decision-log.md §3.6): its backup path is the database. Skills are re-materialised from their rows at every boot, and the files of command-line connections (kubeconfigs, SSH keys) are written again from the secret store the next time the connection is used, so a lost volume is replaced by a new empty one and a control-plane start. The one thing it takes with it is the per-tenant DuckDB lake, which is re-collected from its sources by the collection specs; the pilot tenant has no lake specs, so it has no lake to lose.
The volume is still credential-bearing: those connection files are plaintext copies (mode 0600, under /data/clifiles), named by content and never pruned, so a rotated kubeconfig or key stays next to its successor; an install that predates the database secret store also keeps its old secrets.json.migrated there. Treat the volume, and any copy of it, like the credentials themselves.
Maintenance windows (demo cluster)
ArgoCD denies automated syncs of the platform's two Applications Mon–Fri 09:00–10:30 UTC (infrastructure/gitops/apps/aiops/project.yaml; a manual Sync stays allowed for hotfixes; the cluster's other Applications are not held). Karpenter starts no voluntary node replacement (drift, consolidation) in the same window and replaces one node at a time otherwise; from 6 October 2026 until the hold is removed after the handover it replaces no node for drift or under-use at all, only empty ones (infrastructure/gitops/apps/nodeclass/nodeclass.yaml). Two replacements ignore every budget: an unhealthy node, and expiry — every node expires 14 days after it was created (expireAfter: 336h). A node being replaced keeps pods that a PodDisruptionBudget protects for up to 24 hours. Rotating the control plane's own node moves the control plane: a Recreate, about a minute when a node with room exists in the zone of its /data volume, longer when one has to start. Customer installs have no ArgoCD and their upgrade is a pipeline run scheduled by you; their nodes follow the EKS Auto Mode built-in node pool's own replacement schedule.
Multi-replica control plane: what blocks it
Leader election already keeps the scheduler, audit-chain sealer, job sweep, runner provisioner, the private-chat reconcile sweep and DB gauges on one elected replica (chat turns themselves run on whichever replica serves the request), and the chart refuses replicaCount>1 without it. The platform is still single-replica because the /data volume is ReadWriteOnce (pins every replica to one node); the runner-token rate limit, ingest token bucket and admission gate, SSE pub/sub, chat turn locks and the licence/price caches are per-process memory; three check-then-write paths reachable from any replica are guarded only by in-process locks — IdP email-domain claims (one email domain, one workspace's SSO: two replicas could send a domain's sign-ins to the wrong workspace), compliance incident landing (twin incidents) and change-harvest recording (twin change rows); and every replica adds a connection pool (5 + 10 overflow + the elector's session) that must fit max_connections (three replicas exhausted the sandbox on 3 September 2026). A pooler in transaction mode is no way out: the elector's lock and the online index builder's lock are session locks, and CREATE INDEX CONCURRENTLY refuses to run inside a transaction, so both need a direct connection. Order of work after the 16 October handover: counters to Postgres → IdP domain claims on a normalised table with a unique index, and Postgres advisory locks or partial unique indexes for compliance landing and harvest recording → lake off the PVC and /data to emptyDir → SSE over LISTEN/NOTIFY → pool sizing or a pooler → replicaCount: 2 with anti-affinity and a PDB on the demo first. Decision record: decision-log.md §3.5.
Related: customer-deploy-runbook.md §3 Upgrade and §4 Rollback; architecture.md §11 and §14.