Backup, restore and disaster recovery

Plan for 5–10 minutes of potential data loss with point-in-time restore (PITR). The customer-template drill reached a verified database copy in 21 minutes 56 seconds; this excludes service cutover.

How-to · Updated 2026-10-19 · Describes release 0.1.164

Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).

Plan for 5–10 minutes of potential data loss with point-in-time restore (PITR). The customer-template drill reached a verified database copy in 21 minutes 56 seconds; this excludes service cutover. The control plane stayed on its source throughout both published drills.

What is protected

Daily posture snapshots introduced in 0.1.110 are PostgreSQL rows and are covered by the database backups; they do not depend on /data. This storage coverage is not a separate restore-drill result.

The decision for /data avoids extra plaintext credential copies in volume snapshots. Its rebuild is code-backed, not yet drilled. Revisit the lake decision when a workspace depends on retained lake history that sources cannot re-create.

Recovery objectives and measured evidence

The 7 October record and 8 October record checked identical row counts at the restore point and complete audit-chain replay. The latter checked both chains on the customer-template copy. Each copy was deleted afterwards. These receipts prove copy restoration and integrity, not cutover, RUN-02 end-to-end recovery, a /data rebuild or decryption of every stored connection credential.

Restore a side copy for verification

The following AWS operations quote the tested How to repeat procedure in the 7 October record, also used on 8 October. Example names replace the record's environment names. Run as an approved cloud operator in the source account and region. Set AIOPS_DB_SUBNET_GROUP and AIOPS_DB_SECURITY_GROUP to the source's private groups and AIOPS_RESTORE_TIME to its recorded latest restorable UTC timestamp. The target must be a new drill name, never the source instance name.

AIOPS_SOURCE_DB=aiops-example-org-pg
AIOPS_COPY_DB=aiops-example-org-pg-drill
aws rds describe-db-instances --db-instance-identifier "$AIOPS_SOURCE_DB" \
  --query 'DBInstances[0].LatestRestorableTime'

Read Capture a defensible source baseline and prepare its program before proceeding. Record the timestamp and lag, then capture source-counts.json immediately after selecting T and before the restore request, with the quiet-window evidence described below. Then create a single-AZ verification copy, as the drills did; this is not the Multi-AZ production-cutover recipe.

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier "$AIOPS_SOURCE_DB" \
  --target-db-instance-identifier "$AIOPS_COPY_DB" \
  --restore-time "$AIOPS_RESTORE_TIME" --db-instance-class db.t4g.medium \
  --no-multi-az --db-subnet-group-name "$AIOPS_DB_SUBNET_GROUP" \
  --vpc-security-group-ids "$AIOPS_DB_SECURITY_GROUP" \
  --no-publicly-accessible --enable-iam-database-authentication \
  --no-deletion-protection
aws rds wait db-instance-available --db-instance-identifier "$AIOPS_COPY_DB"
aws rds modify-db-instance --db-instance-identifier "$AIOPS_COPY_DB" \
  --manage-master-user-password --apply-immediately

PostgreSQL PITR rejects --manage-master-user-password on the restore call. The separate modify call gives the copy its own managed password; do not read or reuse the source's master password. Confirm the copy's encryption, parameter group and private network settings against the source before use.

Capture a defensible source baseline

WHERE timestamp <= T is not time travel. A source read made after T cannot reconstruct rows deleted since T, changed timestamps, or the historic tenant population. Do not claim that current source counts equal a historic restore point merely because they are filtered by T.

Before selecting T, agree and record a quiet interval with the operations owner. It must cover T through source-baseline capture, with no deletion, timestamp edit, tenant change or other change to the counted population. Record how writers were controlled and the evidence confirming this; an operator's attestation is not automatically proved by the verifier. Pausing agents alone does not create a quiet interval: intake and other writes can continue. Allow existing audit work to seal before that interval so the set of chain heads is established. Keep the source operating on its original database throughout; this is a read-only comparison, not a service cutover.

Alternatively use an independently retained, approved baseline whose counts and chain membership describe exactly T. Unknown or changed baseline = inconclusive, not a backup failure or a successful restore check. Select another approved window or obtain the required baseline before acceptance.

The program below adapts the recorded drill's five SELECT queries and full verify_chain replay. Its endpoint, baseline and exit guards are additional documentation safeguards, checked locally without database calls. Save it as dr-verify.py on the operator workstation. It does not run application startup, migrations, a grant or a write query. The source mode uses the live pod's existing IAM login; the copy mode uses only the copy's new credential.

import asyncio
import json
import os
import sys
from datetime import datetime, timezone
from urllib.parse import urlsplit

Q = {
    "auditevent": "select count(*) from auditevent where at <= :t",
    "incident": "select count(*) from incident where opened_at <= :t",
    "investigation": "select count(*) from investigation where started_at <= :t",
    "changerecord": "select count(*) from changerecord where created_at <= :t",
    "tenant": "select count(*) from tenant",
}


def compare(source, copy):
    if source.get("baseline_confirmed") is not True:
        return 2, "INCONCLUSIVE: source baseline has not been validated for T"
    if source["restore_time"] != copy["restore_time"]:
        return 1, "FAIL: restore timestamps differ"
    if source["db"][0] != "aiops" or copy["db"][0] != "aiops":
        return 1, "FAIL: unexpected database"
    if source["endpoint"] == copy["endpoint"] or source["db"][1] == copy["db"][1]:
        return 1, "FAIL: source and copy are not distinct"
    if any(type(source[k]) is not int or type(copy[k]) is not int for k in Q):
        return 1, "FAIL: a count is missing or invalid"
    if any(source[k] != copy[k] for k in Q):
        return 1, "FAIL: count mismatch; investigate baseline validity before blaming backups"
    if set(source["chains"]) != set(copy["chains"]):
        return 1, "FAIL: chain membership mismatch"
    results = copy.get("verify", {})
    if not results or set(results) != set(copy["chains"]):
        return 1, "FAIL: no complete chain replay results"
    for chain, result in results.items():
        if not (result.get("ok") and result.get("head_ok")
                and result.get("first_bad_seq") is None
                and result.get("checked") == result.get("last_seq")
                and result.get("last_seq") == copy["chains"][chain]):
            return 1, "FAIL: chain replay or head failed"
    return 0, "PASS: validated baseline counts and every copy chain match"


async def read_database():
    from sqlalchemy import text
    from app.config import settings
    from app.db import SessionLocal
    from app.aiops.audit_chain import verify_chain

    mode = os.environ["DRILL_MODE"]
    if mode not in ("source", "copy"):
        raise ValueError("mode")
    stamp = datetime.fromisoformat(os.environ["DRILL_T"].replace("Z", "+00:00"))
    if stamp.tzinfo is None:
        raise ValueError("restore timestamp must include timezone")
    stamp = stamp.astimezone(timezone.utc)
    endpoint = urlsplit(settings.db_url).hostname
    if endpoint != os.environ["DRILL_EXPECTED_ENDPOINT"]:
        raise ValueError("unexpected endpoint")
    async with SessionLocal() as session:
        db = list((await session.execute(text(
            "select current_database(), host(inet_server_addr()), current_user"
        ))).one())
        if db[0] != "aiops" or not db[1]:
            raise ValueError("unexpected database/server")
        if mode == "copy" and (db[1] == os.environ["DRILL_SRC_ADDR"]
                               or db[2] != os.environ["DRILL_MASTER"]):
            raise ValueError("copy identity mismatch")
        out = {"db": db, "endpoint": endpoint, "restore_time": stamp.isoformat(),
               "baseline_confirmed": mode == "source"
               and os.environ.get("DRILL_BASELINE_CONFIRMED") == "yes"}
        for name, query in Q.items():
            out[name] = (await session.execute(
                text(query), {"t": stamp.replace(tzinfo=None)}
            )).scalar_one()
        out["chains"] = dict((await session.execute(text(
            "select chain, last_seq from auditchainhead"
        ))).all())
        if mode == "copy":
            out["verify"] = {}
            for chain in out["chains"]:
                result = await verify_chain(session, chain, window=None)
                out["verify"][chain] = {key: result.get(key) for key in (
                    "ok", "checked", "first_bad_seq", "head_ok", "last_seq", "unsealed"
                )}
            if not out["verify"] or any(not r["ok"] for r in out["verify"].values()):
                print(json.dumps(out))
                return 1
        print(json.dumps(out))
        return 0


def main():
    try:
        if len(sys.argv) == 3:
            with open(sys.argv[1]) as stream:
                source = json.load(stream)
            with open(sys.argv[2]) as stream:
                copy = json.load(stream)
            code, message = compare(source, copy)
            print(message)
            return code
        return asyncio.run(read_database())
    except Exception as error:
        # Never print a connection URL, password or driver exception text.
        print("FAIL: " + type(error).__name__, file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

Use the same T for both reads. Set AIOPS_SOURCE_ENDPOINT to the source RDS endpoint from its instance details. Set DRILL_BASELINE_CONFIRMED=yes only after the quiet-window/retained-baseline check above; leaving it unset makes final comparison inconclusive. The following source read is the recorded kubectl exec method, with the new guards and a saved JSON result:

kubectl -n aiops exec -i deploy/aiops -c aiops -- \
  env DRILL_MODE=source DRILL_T="$AIOPS_RESTORE_TIME" \
  DRILL_EXPECTED_ENDPOINT="$AIOPS_SOURCE_ENDPOINT" \
  DRILL_BASELINE_CONFIRMED="${DRILL_BASELINE_CONFIRMED:-no}" \
  python - < dr-verify.py > source-counts.json

Keep the JSON and quiet-window evidence in the drill record. The source query uses its existing aiops IAM user and only SELECTs. It contains counts, server address and full opaque chain ids; treat it as operating evidence, not a public example. Do not use a current source JSON as a substitute for an invalid historical baseline.

Run the copy verifier securely

Use the approved platform image: the 8 October receipt used 0.1.108. Capture the actual image reference from the source Deployment, including registry/tag or digest; both reads use that application code. The copy must be reachable from the verification pod through the private subnet/security group route. Do not make the database public to run this check.

In the approved account/region and Kubernetes context, read only the copy's managed-secret ARN, require a distinct endpoint, and stream its URL-encoded credential into a short-lived Kubernetes Secret. Never enable shell tracing, print the URL, write the credential to a local file or use the source master credential. The JSON-to-Secret stream avoids putting the password in kubectl --from-literal process arguments.

set -euo pipefail
set +x
: "${AIOPS_SOURCE_ENDPOINT:?set the source RDS endpoint}"
: "${AIOPS_COPY_DB:?set the disposable copy instance name}"
COPY_ENDPOINT=$(aws rds describe-db-instances --db-instance-identifier "$AIOPS_COPY_DB" \
  --query 'DBInstances[0].Endpoint.Address' --output text)
test -n "$COPY_ENDPOINT" && test "$COPY_ENDPOINT" != "$AIOPS_SOURCE_ENDPOINT"
COPY_SECRET_ARN=$(aws rds describe-db-instances --db-instance-identifier "$AIOPS_COPY_DB" \
  --query 'DBInstances[0].MasterUserSecret.SecretArn' --output text)
COPY_MASTER=$(aws rds describe-db-instances --db-instance-identifier "$AIOPS_COPY_DB" \
  --query 'DBInstances[0].MasterUsername' --output text)
VERIFY_IMAGE=$(kubectl -n aiops get deploy aiops \
  -o jsonpath='{.spec.template.spec.containers[0].image}')
SOURCE_ADDR=$(python3 -c 'import json; print(json.load(open("source-counts.json"))["db"][1])')
aws secretsmanager get-secret-value --secret-id "$COPY_SECRET_ARN" \
  --query SecretString --output text | COPY_ENDPOINT="$COPY_ENDPOINT" python3 -c '
import json, os, sys
from urllib.parse import quote
d = json.load(sys.stdin)
url = "postgresql+asyncpg://%s:%s@%s:5432/aiops" % (
    quote(d["username"], safe=""), quote(d["password"], safe=""), os.environ["COPY_ENDPOINT"])
json.dump({"apiVersion": "v1", "kind": "Secret", "metadata": {
    "name": "dr-drill-db", "namespace": "aiops"}, "type": "Opaque",
    "stringData": {"AIOPS_DB_URL": url}}, sys.stdout)
' | kubectl -n aiops create -f - >/dev/null
kubectl -n aiops create configmap dr-drill-program --from-file=dr-verify.py

Use fresh dr-drill-* names; stop if they belong to another operation. The following pod packages the recorded non-root, no-service-links copy read with the program mounted as a ConfigMap. Its metadata, mount and exit-result collection are documentation wrappers, not a newly measured live receipt. The Secret belongs only to the disposable copy. This deliberately omits the drill's unused conditional GRANT: a missing read privilege fails the check and needs a separately approved operator decision.

VERIFY_IMAGE="$VERIFY_IMAGE" COPY_ENDPOINT="$COPY_ENDPOINT" COPY_MASTER="$COPY_MASTER" \
  SOURCE_ADDR="$SOURCE_ADDR" DRILL_T="$AIOPS_RESTORE_TIME" python3 - <<'PY' | kubectl -n aiops create -f -
import json, os, sys
env = [{"name": key, "value": value} for key, value in {
    "DRILL_MODE": "copy", "DRILL_T": os.environ["DRILL_T"],
    "DRILL_EXPECTED_ENDPOINT": os.environ["COPY_ENDPOINT"],
    "DRILL_SRC_ADDR": os.environ["SOURCE_ADDR"], "DRILL_MASTER": os.environ["COPY_MASTER"],
    "AIOPS_DB_IAM_AUTH": "false",
}.items()]
env.append({"name": "AIOPS_DB_URL", "valueFrom": {"secretKeyRef": {
    "name": "dr-drill-db", "key": "AIOPS_DB_URL"}}})
json.dump({"apiVersion": "v1", "kind": "Pod", "metadata": {
    "name": "dr-drill-verify", "namespace": "aiops"}, "spec": {
    "restartPolicy": "Never", "enableServiceLinks": False,
    "automountServiceAccountToken": False,
    "securityContext": {"runAsNonRoot": True, "runAsUser": 1000},
    "containers": [{"name": "verify", "image": os.environ["VERIFY_IMAGE"],
        "command": ["python", "-c", "exec(compile(open('/verify/dr-verify.py').read(), '/verify/dr-verify.py', 'exec'))"], "env": env,
        "securityContext": {"allowPrivilegeEscalation": False, "capabilities": {"drop": ["ALL"]}},
        "volumeMounts": [{"name": "program", "mountPath": "/verify", "readOnly": True}]}],
    "volumes": [{"name": "program", "configMap": {"name": "dr-drill-program"}}],
}}, fp=sys.stdout)
PY

The python -c entrypoint keeps the image's /srv working directory on its module search path, as in the recorded drill, so the mounted program can import app without a package install or application startup.

Check termination and require exit code 0 before comparing; a container waiting/running is not completion. Capture logs only into the private evidence file and run the local, database-free comparison:

kubectl -n aiops get pod dr-drill-verify \
  -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
kubectl -n aiops logs dr-drill-verify > copy-counts.json
python3 dr-verify.py source-counts.json copy-counts.json

The program emits every full chain id from auditchainhead and its complete replay result; no eight-character truncation can merge two results. For every copy chain require ok, head_ok, first_bad_seq: null and checked == last_seq == recorded copy head. Keep reported unsealed rows visible in the receipt. The final comparison also requires the validated baseline's chain membership and five integer counts, same T/database and different source/copy endpoint and server address. It does not require the later source head sequence to equal the earlier copy's head.

Exit 0 means these bounded acceptance checks passed. Exit 1 is a query, identity, count or replay failure; exit 2 means baseline inconclusive. Only a completed pod with exit 0 and local comparison exit 0 can be recorded as a passed copy check. Investigate mismatches without assuming a post-T source mutation means a defective backup.

Delete the verifier resources even after an unsuccessful read. Preserve sanitized failure results first, then confirm these names belong to this drill:

kubectl -n aiops delete pod dr-drill-verify --ignore-not-found
kubectl -n aiops delete configmap dr-drill-program --ignore-not-found
kubectl -n aiops delete secret dr-drill-db --ignore-not-found

After recording successful verification, confirm AIOPS_COPY_DB is still the drill copy, then use the tested cleanup:

aws rds delete-db-instance --db-instance-identifier "$AIOPS_COPY_DB" \
  --skip-final-snapshot --delete-automated-backups

Record deletion completion and removal of its managed secret. Keep the source untouched. A failed verification is a failed drill; preserve the sanitized evidence and investigate before accepting backups as usable.

Recover service and run the monthly check

A real recovery additionally needs Rollback, data restore: database login authorization for the new instance, controlled name swap, control-plane restart and Terraform state reconciliation before another apply. Use the production settings in deployment runbook §4, not the single-AZ disposable-copy flags above. If restoring across a key rotation, deploy the retained key as AIOPS_TENANT_SECRET_KEY_PREVIOUS so startup can re-key the restored secrets. Verify connection decryption and execution separately, and replay every audit chain after restore.

Monthly, run the side-copy check in an approved window and retain source backup settings, restore point, lag, available/verified times, count comparison, chain outcomes, key-retention check and cleanup receipt. Include service cutover and connection-decryption results only when actually tested. The current cutover/state-import gap remains open.

Sources: Availability, Deployment runbook §4 and §12.1, audit retention after restore, and the two linked drill records.

Next steps

• Next: Platform monitoring and alerting
• All Install and operate the platform pages
• Documentation home