# Architecture — AIOps Control Deck

> The diagram set: system context, components, the closed loop, the key sequences and the state machines, all generated from the code.

_Concept · Updated 2026-10-24 · Describes release 0.1.164_

Software design reference for the AIOps platform POC. Narrative + decision record live in ../DESIGN.md; the data model lives in db-schema.md. This file is the **diagram set**: system context, components, the closed loop, the key sequences, and the state machines — all generated from the code (`app/aiops/*`, `app/api/aiops.py`, `app/models.py`).

Every diagram is Mermaid and renders in the Azure DevOps repo/wiki preview and on GitHub.

---

## 1. System context {#1-system-context}

Who and what touches the platform.

```mermaid
flowchart TB
    subgraph actors["People"]
        sre["SRE / Operator<br/>(tenant member)"]
        approver["Approver<br/>(signs off remediations)"]
        clevel["Executive / C-level<br/>(insights chat — read-only,<br/>builtin executive role)"]
        staff["Platform staff<br/>(superadmin — NO tenant data<br/>without support grant)"]
    end

    subgraph sources["Signal sources (per tenant)"]
        am["Alertmanager"]
        cw["CloudWatch → SNS"]
        azm["Azure Monitor"]
        graf["Grafana / Datadog"]
    end

    platform["**AIOps Control Deck**<br/>FastAPI · Jinja/HTMX UI · MCP servers"]

    subgraph ext["External services"]
        llm["LLM provider<br/>(litellm: Bedrock Nova /<br/>OpenAI-compatible)"]
        clouds["Cloud + observability APIs<br/>(AWS/Azure/GCP, Prometheus,<br/>Grafana, k8s via MCP)"]
        git["Git host<br/>(GitHub / GitLab / Azure Repos)"]
        idp["OIDC IdPs<br/>(platform issuer + per-tenant<br/>TenantIdP: Entra / Keycloak realm)"]
    end

    db[("Postgres / SQLite")]
    sec[["Encrypted secret store<br/>(Fernet)"]]
    lakedb[("Per-tenant DuckDB lakehouse<br/>(PII-scrubbed telemetry,<br/>entitlement-gated)")]

    sources -- "webhook (token)" --> platform
    sre -- "portal / REST / MCP" --> platform
    approver -- "approve · deny · edit" --> platform
    clevel -- "ask (chat.ask perm)" --> platform
    staff -- "admin console + audited support access" --> platform

    platform -- "completion + tool calls" --> llm
    platform -- "read (signed) / gated write" --> clouds
    platform -- "scheduled telemetry pulls" --> clouds
    platform -- "gitops_pr = open PR, never touch live" --> git
    platform -- "login — issuer discovered<br/>by email domain" --> idp
    platform --- db
    platform --- sec
    platform --- lakedb
```

## 2. Components (containers within the single app) {#2-components-containers-within-the-single-app}

One FastAPI process. `app/aiops/runtime.install()` is the single wiring seam that connects ingest → engine → API/MCP/scheduler (it wires module-level `ENGINE_STARTER`/`ENGINE_RESUMER` seams — it takes no arguments).

```mermaid
flowchart LR
    subgraph edge["HTTP surface"]
        webhook["ingest.py<br/>/webhook/{source}"]
        rest["api/aiops.py<br/>/api/aiops/*"]
        ui["ui_aiops.py + Jinja<br/>/portal · /admin"]
        mcpt["mcp_tenant.py<br/>/mcp/aiops"]
        mcpa["mcp_admin.py<br/>/mcp/admin"]
    end

    subgraph brain["Agent brain (our IP)"]
        runtime["runtime.py<br/>STARTER / RESUMER seams<br/>+ per-tenant semaphore"]
        engine["engine.py<br/>tool-calling loop + HITL pause"]
        pillars["pillars.py<br/>(tenant,pillar) build factory"]
        routing["models_routing.py<br/>tier map → model · ladder ·<br/>escalation triggers · suggestions"]
        tools["tools.py<br/>connectors → tools<br/>(api/cli/mcp + cloud natives)"]
        remediate["remediate.py<br/>execute_route() (route button)"]
        memory["memory.py<br/>feedback → SkillProposal"]
        scheduler["scheduler.py<br/>cron pillars + TTL sweeps<br/>+ lake collect/retention ticks"]
        collector["collector.py<br/>CollectionSpec pull loop<br/>(feature-gated)"]
        pii["pii.py<br/>PII scrubber<br/>(salted pseudonyms)"]
        chat["chat.py<br/>insights chat<br/>(perm-filtered registry)"]
        xray["xray.py + grounding.py + xray_ledger.py<br/>estate x-ray mission loop<br/>(read-only wrapped, budgeted, grounded)"]
        artifacts["artifacts.py<br/>create_artifact chokepoint<br/>(scrub + safe_url on write)"]
        lakehouse[("lake.py<br/>per-tenant DuckDB<br/>bronze + silver")]
    end

    subgraph authz["Auth & tenancy"]
        auth["auth.py + oidc_bearer.py<br/>session / PAT / OIDC bearer<br/>(multi-issuer: global + TenantIdP)"]
        az["aiops/authz.py<br/>get_tenant_perms · require_perm<br/>(disabled tenant fails closed)"]
        secrets["secrets.py<br/>Fernet secret store"]
        kcp["aiops/kc_provision.py<br/>realm-per-tenant SSO provisioning<br/>(superadmin one-click)"]
    end

    db[("DB")]
    stream["stream.py<br/>in-proc SSE pub/sub"]
    llm["litellm"]
    kc["Keycloak admin API<br/>(AIOPS_KEYCLOAK_ADMIN_URL)"]

    webhook --> runtime
    rest --> az
    ui --> az
    mcpt --> az
    mcpa --> auth
    rest --> runtime
    mcpt --> runtime
    scheduler --> runtime
    runtime --> engine
    engine --> pillars
    engine --> tools
    pillars -- "tier → model resolve" --> routing
    runtime -- "pre-run escalation" --> routing
    rest -- "tier suggest / discovery" --> routing
    engine --> stream
    runtime --> secrets
    remediate --> secrets
    remediate --> tools
    rest --> remediate
    rest --> memory
    rest --> chat
    rest --> xray
    xray --> engine
    xray -- "report artifact" --> artifacts
    xray -- "forced scrub" --> pii
    remediate -- "PR artifact (via rest)" --> artifacts
    scheduler --> collector
    collector --> pii
    collector --> lakehouse
    chat --> engine
    chat -- "final egress scrub" --> pii
    engine -- "lake_query (read-only)" --> lakehouse
    engine --> llm
    rest -- "provision-keycloak" --> kcp
    kcp --> kc
    edge --> db
    brain --> db
    az --> db
    stream --> rest
```

## 3. The closed loop (pipeline) {#3-the-closed-loop-pipeline}

```mermaid
flowchart TB
    A["Alerts fire<br/>Alertmanager · CloudWatch · Azure Monitor · Datadog"]
    B["**ingest.py** — adapter normalizes to canonical labels<br/>dedupe (tenant+fingerprint)"]
    C{"correlate:<br/>shared key within<br/>window?"}
    D["join existing Incident<br/>child alert **suppressed**<br/>(the 300→1 story)"]
    E["new **Incident** + **Investigation**<br/>route pillar (security-ish → secops)"]
    F["**engine.py** — agentic loop<br/>LLM ⇄ tools from tenant datasources"]
    G{"tool needs<br/>write / approval?"}
    H["**ApprovalRequest** — loop PAUSES<br/>incident = awaiting_approval"]
    I["human: approve · deny · edit<br/>(portal / REST / MCP)"]
    X["**engine.py** executes the approved tool<br/>ToolRegistry.invoke → incident remediating"]
    O["**engine wrap-up** → Investigation.rca_markdown<br/>(LLM structured RCA — persisted)"]
    P["**memory.py** — feedback → SkillProposal<br/>promote → SKILL.md → next run loads it"]

    OP["operator: **Remediate via route** button<br/>(POST .../remediate — the click IS the approval)"]
    J["**remediate.py** execute_route()<br/>first RemediationRoute by priority"]
    M{"route<br/>matched?"}
    K["cli · api · mcp<br/>(throwaway ToolRegistry)"]
    L["**gitops_pr**<br/>branch + commit + PR<br/>live infra NEVER touched"]
    N["**escalate to human**<br/>(never guess a write path)"]

    A --> B --> C
    C -- yes --> D
    C -- no --> E
    D --> F
    E --> F
    F --> G
    G -- no (read) --> F
    G -- yes --> H --> I
    I -- approve/edit --> X --> F
    I -- deny --> F
    F -- wrap-up --> O

    OP --> J --> M
    M -- yes --> K
    M -- yes --> L
    M -- no --> N
    K --> O
    L --> O
    N --> O
    O --> P
    P -. "feeds the next investigation" .-> F
```

Two distinct write paths: **(a)** a gated tool the agent proposed mid-loop is executed by the engine itself on approval (`engine.resume` → `ToolRegistry.invoke`); **(b)** the operator-triggered **Remediate via route** button (`POST .../incidents/{id}/remediate`) runs `remediate.execute_route()` — the RemediationRoute channels (cli/api/mcp/gitops_pr), where the button click is itself the human approval. `rca.py` is a deterministic row-backed RCA renderer that ships in the repo but is **not yet wired** — today the persisted `Investigation.rca_markdown` is the engine's LLM wrap-up.

Alert resolution auto-closes an incident only from `open`/`investigating` — never from `awaiting_approval`/`remediating`, where a human holds the loop (`ingest.py`).

## 4. Sequence — alert storm → RCA (the demo path) {#4-sequence-alert-storm-rca-the-demo-path}

```mermaid
sequenceDiagram
    autonumber
    participant Src as Alert source
    participant IN as ingest.py
    participant DB as DB
    participant RT as runtime.py
    participant EN as engine.py
    participant LLM as litellm
    participant SSE as stream.py (SSE)
    participant H as Human (portal)
    participant RM as remediate.py
    participant Cloud as Cloud / Git

    Src->>IN: POST /webhook/{source}/{tenant_slug}/{token}<br/>(token via path or Authorization header)
    Note over IN: adapter → canonical labels,<br/>dedupe by fingerprint
    IN->>DB: 5 Alerts, correlate → 1 Incident<br/>(4 suppressed) + 1 Investigation
    IN-->>Src: 200 {incidents_created, suppressed, ...}
    IN->>RT: ENGINE_STARTER(inv_id) [detached task,<br/>per-tenant semaphore]

    RT->>EN: run(tenant, pillar, incident)
    loop until answer or max_steps
        EN->>LLM: messages + tool schemas
        LLM-->>EN: tool call (e.g. prometheus_query)
        EN->>Cloud: read (signed / gated)
        Cloud-->>EN: result (truncated)
        EN->>SSE: publish tool event
        EN->>DB: persist conversation + tool_calls
    end

    Note over EN: proposes a WRITE (restart / PR)
    EN->>DB: ApprovalRequest(pending)<br/>incident → awaiting_approval
    EN->>SSE: publish approval event
    EN-->>RT: paused_approval

    H->>DB: POST .../investigations/{id}/decisions (approve)
    DB->>RT: ENGINE_RESUMER(inv_id)
    RT->>EN: resume — execute the APPROVED TOOL directly
    EN->>Cloud: approved write (ToolRegistry.invoke)
    Cloud-->>EN: result → incident remediating
    EN->>DB: RCA wrap-up (Investigation.rca_markdown)<br/>incident → resolved / escalated
    EN->>SSE: publish done

    opt operator uses the Remediate-via-route button (separate path)
        H->>RM: POST .../incidents/{id}/remediate → execute_route()
        alt route matches
            RM->>Cloud: gitops_pr → open PR (no live change)
            RM-->>H: {ok, artifact: PR url}
        else no match
            RM-->>H: escalate
        end
    end

    H->>DB: 👎 + correction → Feedback → SkillProposal
```

**Model tier routing inside this sequence** (`models_routing.py` — deterministic, our code decides, never an LLM): the loop calls run on the tenant's `standard`-tier model and the RCA wrap-up on `reasoning` (a `PillarAgent.model` override pins both and disables switching). A `severity=critical` or prod-labelled incident starts the loop on the reasoning tier — decided once at start; resume does NOT re-derive it, the recorded `routing:` system note in the conversation re-applies the switch. Mid-run, 3 byte-identical tool calls or 3 parse-failure turns climb one configured ladder step, and at 70% of the cost cap the loop drops one (once each per run; wrap-up never downgraded, and after a budget downgrade escalation stays off — budget beats escalation). The switch note is inserted at the top of the loop iteration, so it never splits an assistant `tool_calls` message from its tool results; every switch also lands a `model.escalated`/`model.downgraded` audit row, and per-model spend accumulates in `Investigation.model_usage`.

Structured wrap-ups request a finite 32,768-token completion allowance, including provider reasoning usage, clamped to a smaller known model output limit from the loaded catalogue. A provider `length` finish reason permits one concise, complete rewrite with the same schema and no tools, subject to cancellation and the run's remaining budget. Both responses are accounted for and their finish reasons recorded. Partial JSON is never accepted as a remediation plan.

## 5. Sequence — auth + RBAC permission resolution {#5-sequence-auth-rbac-permission-resolution}

Three credential types collapse to a `User`; then a per-tenant **permission set** decides the call. No superadmin bypass: staff without a membership get **404** on tenant data.

```mermaid
sequenceDiagram
    autonumber
    participant C as Client (browser / MCP / curl)
    participant A as auth.py (current_user)
    participant Z as aiops/authz.py
    participant DB as DB

    alt browser
        C->>A: session cookie
        A->>DB: AuthSession by sha256(token)
    else PAT
        C->>A: Authorization: Bearer ntp_...
        A->>DB: ApiToken by sha256 → User (+ scope cap)
    else OIDC
        C->>A: Bearer <JWT>
        A->>A: verify iss+aud/azp+exp+RS256 (oidc_bearer.py)
        A->>DB: User by oidc_sub only (bearer never auto-links —<br/>email linking happens in the login flow, not here)
    end
    A-->>Z: User (or 401)

    Z->>DB: TenantMember(user, tenant)?
    alt no membership (incl. plain superadmin)
        Z-->>C: 404 (existence hidden)
    else membership (role_name → TenantRoleDef, or legacy enum)
        Z->>DB: resolve permission set (expired row → treated absent)
        alt perm present
            Z-->>C: proceed
        else perm missing
            Z-->>C: 403 (requires <perm>)
        end
    end

    Note over C,DB: Staff break-glass: POST /{tenant}/support-access →<br/>audited, expiring TenantMember(support=true) the customer can see + revoke
```

### 5.1 Sequence — per-tenant SSO login (Company SSO) {#5-1-sequence-per-tenant-sso-login-company-sso}

A tenant's OWN OIDC issuer (`TenantIdP` — Entra app, Keycloak realm, Google…) signs its users in. The login page discovers the IdP by email domain; the callback enforces the hardening invariants (no account takeover, `email_verified` always, no `members.manage` via SSO — see DESIGN.md §4).

```mermaid
sequenceDiagram
    autonumber
    participant U as User (browser)
    participant P as api/auth.py
    participant IdP as Tenant IdP (own issuer)
    participant DB as DB

    U->>P: POST /oidc/discover (email)
    P->>DB: enabled TenantIdP claiming the email's domain?
    P-->>U: {found, idp_id, provider_name}
    U->>P: GET /oidc/login?idp=(idp_id)
    P->>IdP: discovery doc (cached per issuer)
    P-->>U: 303 to authorize (state + nonce + PKCE — idp rides the HMAC-signed cookie)
    U->>IdP: authenticate with company credentials
    IdP-->>U: 303 back with code
    U->>P: GET /oidc/callback
    P->>IdP: exchange code + fetch userinfo (client secret from the tenant secret store)
    Note over P: email_verified ALWAYS required (tenant IdPs ignore the global relax toggle)<br/>email domain must be in the IdP's declared email_domains<br/>sub stored ISSUER-SCOPED — issuer|sub (User.oidc_sub)
    alt account with this email exists
        Note over P,DB: auto-link ONLY if _tenant_link_ok — never a superadmin,<br/>local-password, cross-tenant or non-oidc account (no takeover) —<br/>else refuse with "ask a tenant owner to invite you"
    else no account (auto_provision on)
        P->>DB: create User(auth_source=oidc)
    end
    Note over P: superadmin promotion (oidc_superadmin_emails) NEVER applies on this path
    P->>DB: auto-provision TenantMember — group_role_map (first sorted match) else default_role<br/>existing member never overwritten
    Note over P,DB: a role carrying members.manage is REFUSED at login —<br/>the backstop behind the config-time gates (_check_idp_roles on PUT + provision,<br/>_reject_if_idp_referenced on role create/edit)
    P-->>U: session cookie (a disabled tenant refuses — and fails closed at authz too)
```

The **bearer** path (`aiops/oidc_bearer.py`) mirrors the issuer selection: the token's `iss` picks the global issuer or an enabled `TenantIdP`, that issuer's JWKS verifies RS256 and its client_id is checked against `aud`/`azp`, and the sub lookup uses the issuer-scoped form — but bearer auth never auto-creates users or memberships (login-flow only).

## 6. Sequence — multi-cloud connector rendering {#6-sequence-multi-cloud-connector-rendering}

How a `TenantDatasource` becomes an invocable, credential-signed tool. No cloud SDKs — SigV4 / AAD / GCP-JWT are hand-rolled (`tools.py`).

```mermaid
sequenceDiagram
    autonumber
    participant EN as engine.py
    participant TR as ToolRegistry (tools.py)
    participant DT as DatasourceType (catalog)
    participant SS as secret store (Fernet)
    participant Cred as credential broker
    participant API as Cloud API

    EN->>TR: build tools for (tenant, pillar)
    TR->>DT: config_template + fields_schema
    Note over TR,SS: runtime._load_datasources resolves secret_ref via the store,<br/>fills ${field} placeholders — the registry gets already-resolved config
    Note over TR: bind by type_key (aws_request / azure_request /<br/>gcp_request / prometheus_query / <name>_request)

    EN->>TR: invoke aws_request(service, action, ...)
    Note over TR: gate by verb — Describe*/List*/Get*/<br/>GenerateCredentialReport free, everything else → ApprovalRequest
    TR->>Cred: _aws_credentials(cfg)
    alt role_arn present
        Note over Cred: no keys = keyless — signed with the pod's own<br/>workload identity (Pod Identity, else IRSA),<br/>external_id mandatory
        Cred->>API: STS AssumeRole (SigV4)
        API-->>Cred: temp creds (cached to expiry)
    end
    Cred-->>TR: signing creds
    TR->>API: SigV4-signed request
    API-->>TR: response (capped)
    TR-->>EN: tool result

    Note over TR,API: gitops_pr write path (remediate.py) dispatches on provider:<br/>GitHub /contents+/pulls · GitLab /files+/merge_requests ·<br/>Azure Repos /pushes+/pullrequests — same contract, PR is the remediation
```

**Keyless AWS roles.** The External ID is minted per workspace by the platform (`grounding.mint_external_id`), never typed by a user. Which roles the guided CloudShell setup creates depends on the deployment:

- **Staff-prepared** (default): platform staff publish an exact read/write role pair per account, plus the workspace's reader where reads run on runners (`PUT /api/aiops/admin/tenants/{id}/cloud-runtime`), and the setup creates exactly those.
- **Any account** (`AIOPS_KEYLESS_AWS_ANY_ACCOUNT=true`, set in the customer example values since 0.1.109): the control-plane and runner roles may assume `arn:aws:iam::*:role/aiops-*` (`sts:AssumeRole` + `sts:TagSession`), so the setup creates `aiops-<slug>-read`, trusting both roles, and `aiops-<slug>-write`, trusting the control plane only, in any account but the platform's own. That account, a workspace with a private-network agent and one with staff AWS setup keep prepared pairs.

Either way the platform IAM in the customer template grants `sts:AssumeRole` only on roles named `aiops-*` with no path (in the platform's own account a trust that names the control plane suffices), and the staff API refuses a prepared pair named otherwise outside the control plane's account (422). `cloud_setup.aws_setup_gap` is the one readiness model the setup preview, the Test refusal text and the portal read.

## 7. Sequence — telemetry collection → PII scrub → lakehouse {#7-sequence-telemetry-collection-pii-scrub-lakehouse}

The pull loop (`app/aiops/collector.py` + `app/aiops/lake.py`) runs enabled collection specifications for valid enabled workspaces. Lakehouse is included by default; `feature_active()` no longer checks grants or terms. The scrub is structural: `lake.write_rows` has no unscrubbed path.

```mermaid
sequenceDiagram
    autonumber
    participant SCH as scheduler loop (60s tick)
    participant CO as collector.py
    participant DB as DB
    participant DS as Tenant datasource<br/>(prometheus / loki / ES / custom)
    participant PII as pii.py
    participant LK as lake.py (DuckDB)

    SCH->>CO: collect_due()
    CO->>DB: select enabled CollectionSpecs (short session)
    Note over CO,DB: due = interval elapsed AND<br/>feature_active(lakehouse) for the tenant
    loop each due spec
        CO->>DB: stamp last_run_at BEFORE the fetch (commit)
        Note over CO: no DB session held across HTTP —<br/>a slow connector cannot block the app<br/>or cause double-ingest
        CO->>DS: fetch with rendered config + secret store<br/>(SSRF check — the datasource's own hosts allowlisted)
        DS-->>CO: payload (bronze) + normalized silver rows
        CO->>PII: scrub_json every row — MANDATORY<br/>(keys and values, salted pseudonym tokens)
        PII-->>CO: clean rows + findings by kind
        CO->>LK: INSERT bronze_records + metrics/logs
        alt findings > 0
            CO->>DB: upsert compliance Incident<br/>(correlation key pii:{datasource}, deduped while active)
            CO->>DB: AuditEvent pii.detected
        end
        CO->>DB: stamp last_status (ok: N rows / error)
    end

    Note over SCH,LK: retention sweep — same loop, at most daily per tenant:<br/>DELETE rows older than aiops_lake_retention_days (default 30)
```

## 8. Sequence — insights chat turn {#8-sequence-insights-chat-turn}

One turn of the read-only C-level chat (`app/aiops/chat.py`). The registry is filtered by the CALLER's permission set at build time — a member can never read via chat what REST would deny.

```mermaid
sequenceDiagram
    autonumber
    participant U as User (chat.ask perm)
    participant API as api/aiops.py
    participant CH as chat.py
    participant EN as ChatEngine (engine loop)
    participant LK as lake.py
    participant PII as pii.py
    participant DB as DB
    participant SSE as stream.py (SSE)

    U->>API: POST /api/aiops/{t}/chat (message, session_id?)
    API->>DB: ChatSession — create, or load OWN-user only<br/>(another member, superadmin included → 404)
    API-->>U: 202 (session_id) — turn runs DETACHED

    API->>CH: start_chat_turn(sid, message)
    Note over CH: per-session turn lock (busy → 409)<br/>+ the per-tenant concurrency semaphore
    CH->>DB: resolve caller perms + tenant PII salt
    CH->>CH: build registry FILTERED by caller perms<br/>(read-only sys_* tools — a denied tool is simply absent)
    opt lakehouse active AND datasources.view
        CH->>LK: register lake_query + catalog into the prompt
    end
    CH->>DB: Investigation(trigger=chat, status=running)<br/>(hidden from incident APIs/dashboards)
    CH->>EN: run() — the real engine loop<br/>(context budget + token/cost accounting apply)
    loop tool calls
        EN->>DB: sys_* read-only query / lake_query
        EN->>SSE: publish tool event (topic chat:{sid})
    end
    EN-->>CH: final assistant answer
    CH->>PII: scrub_text(answer) — FINAL EGRESS GUARD
    CH->>DB: append turn + tool_trace to ChatSession<br/>+ AuditEvent chat.ask (question HASH, never the text)
    CH->>SSE: publish answer + done

    U->>API: GET /chat/{sid}/stream (SSE)
    API-->>U: snapshot (persisted messages) then live events
    Note over U,API: the snapshot covers the answer-before-connect race —<br/>a turn that finished early is still delivered
```

MCP `chat_ask` takes the same path but runs `run_turn` **synchronously** (the answer comes back in the tool result); the lock, semaphore, egress scrub and audit are inside `run_turn`, so both callers share them.

## 9. Sequence — Estate X-ray scan {#9-sequence-estate-x-ray-scan}

The premium one-shot deep scan (`app/aiops/xray.py`). The phase orchestration is **deterministic code at our layer** — no LLM meta-agent; each phase is one engine run on an ephemeral `Investigation(trigger=xray)` (hidden from incident surfaces, like chat turns). The whole scan is read-only by construction and PII-scrubbed with no opt-out.

```mermaid
sequenceDiagram
    autonumber
    participant U as Operator (datasources.manage)
    participant API as api/aiops.py
    participant XR as xray.py (run_scan)
    participant EN as XrayEngine (engine loop)
    participant DS as Datasource tools<br/>(ReadOnlyRegistry-wrapped)
    participant PII as pii.py
    participant AR as artifacts.py
    participant DB as DB

    U->>API: POST /api/aiops/{t}/xray (datasource_ids, model?/models?, budget_usd?)
    Note over API: perm + feature_active(xray) + budget validation —<br/>positive, FINITE, capped by aiops_xray_max_cost_usd;<br/>running-scan count capped (aiops_xray_max_concurrent)
    API->>DB: XrayScan(running)
    API-->>U: 201 {scan_id} — the scan runs DETACHED
    API->>XR: XRAY_RUNNER(scan_id)

    XR->>DB: load grounding snapshot (catalog, datasources,<br/>knowledge, specs, pillars — never secret values)
    XR->>DB: Investigation(trigger=xray, status=running)
    XR->>EN: ONE mission run under the per-tenant semaphore<br/>(per-step cost cap = REMAINING scan budget;<br/>per-step cancel poll _should_stop)

    loop mission steps — the agent self-directs (aiops_xray_mission_steps cap)
        EN->>DS: read probe — a gated call is DENIED, never executed<br/>(no ApprovalRequest, no pause)
        EN->>PII: scrub tool output (forced on — no settings flag)
        EN->>DB: ledger tools (plan_update / record_finding /<br/>record_pillar_coverage / draft_recommendation) —<br/>scrubbed at capture, persisted to the scan row PER STEP,<br/>drafts VALIDATED against grounding (invalid → rejected into the loop)
        Note over EN: agent stops calling tools → finish-gate _may_finish:<br/>checklist unmet (plan open / ds untouched / pillars missing)<br/>→ nudge back into the loop (≤ aiops_xray_max_nudges, never past 80% budget)
    end

    alt mission completed within budget
        EN-->>XR: small executive-summary JSON (failure tolerated —<br/>the report never depends on it)
        XR->>XR: zero-LLM sweep — map_discoveries (found services →<br/>ready-to-connect suggestions) + deterministic 7-pillar gap table
        XR->>XR: render_report_md from the LEDGER (code, no giant prompt)
        XR->>PII: scrub_text(report) + scrub_json(recommendations)
        XR->>AR: create_artifact(kind=report, source_type=xray)
        XR->>DB: status done + AuditEvent xray.finished (totals)
    else budget hit mid-step
        XR->>AR: partial report FROM THE LEDGER (findings + open plan items)<br/>still becomes an Artifact
        XR->>DB: status aborted_budget + AuditEvent xray.finished
    else cancelled (POST .../cancel) or feature revoked
        XR->>DB: status cancelled — the per-step poll stops the spend<br/>within one step; ledger spend stays recorded
    end

    U->>API: GET /xray/{scan_id} — live poll target<br/>(per-phase tokens/cost + running totals + report)
    U->>API: POST /xray/{scan_id}/apply (kind, index) — per-kind permission
    API->>DB: DRAFT rows only — disabled CollectionSpec/RemediationRoute,<br/>INACTIVE [x-ray]-prefixed KnowledgeItem
    Note over U,DB: HITL activation is the persistence gate —<br/>knowledge goes live only via POST /knowledge/{id}/activate
```

The cancel endpoint flips the row terminal **immediately** (freeing a slot under the running-scan cap); the detached mission notices via its per-step `_should_stop` poll and stops within one step. Any error marks the scan (and a still-running phase Investigation) `failed` — `run_scan` never raises and never leaves a row stuck `running`. The legacy 3-phase flow (`aiops_xray_mission=false`) keeps its old phase-boundary checks unchanged.

## 10. State machines {#10-state-machines}

### Incident FSM {#incident-fsm}

Derived from `app/aiops/models._INCIDENT_TRANSITIONS`. The forward path is the design; the extra edges are legitimate manual/concurrent moves (a deny bounces back; a resumed run pauses on a second gated tool; a human re-triggers a closed incident). `transition()` is the **primary** FSM writer (used by `ingest.py` + `engine.py`) — it audits every move (`incident.status`) and **skips** (never raises) illegal ones. Two operator endpoints, `escalate` and `remediate`, set the status directly with their own audit events (`incident.escalate` / `incident.remediate`), bypassing the guard.

```mermaid
stateDiagram-v2
    [*] --> open: alert / manual
    open --> investigating
    investigating --> awaiting_approval: agent proposes a write
    investigating --> remediating: approved write executes
    awaiting_approval --> remediating: approved
    awaiting_approval --> investigating: denied (bounce)
    remediating --> awaiting_approval: 2nd gated tool
    remediating --> investigating: keep probing
    remediating --> resolved: fix verified
    investigating --> resolved: no action needed
    open --> resolved: alert cleared
    resolved --> [*]

    investigating --> escalated: 20% human path
    awaiting_approval --> escalated
    remediating --> escalated
    open --> escalated
    escalated --> [*]

    resolved --> investigating: human re-triggers
    escalated --> investigating: human re-triggers

    note right of open
        Also legal (not drawn, to keep it readable):
        open may pause straight to awaiting_approval
        or remediating for a manually-created incident.
    end note
```

### ApprovalRequest FSM {#approvalrequest-fsm}

```mermaid
stateDiagram-v2
    [*] --> pending
    pending --> approved: human approves (execute tool)
    pending --> edited: approve w/ modified command (cli only)
    pending --> denied: reject (feedback → LLM)
    approved --> [*]
    edited --> [*]
    denied --> [*]
    note right of pending
        Re-decide is rejected (409): first decision wins.
        Unattended > TTL → denied by the scheduler sweep,
        incident → escalated.
    end note
```

### SkillProposal FSM (the memory rollback path) {#skillproposal-fsm-the-memory-rollback-path}

```mermaid
stateDiagram-v2
    [*] --> draft: feedback / correction
    draft --> active: approve (write SKILL.md, reload catalog)
    draft --> rejected: reject
    active --> retired: retire (delete skill dir — rollback)
    rejected --> [*]
    retired --> [*]
```

## 11. Deployment — compose-first {#11-deployment-compose-first}

```mermaid
flowchart TB
    subgraph host["docker compose (demo laptop)"]
        subgraph core["default services (no profile)"]
            app["aiops (FastAPI)"]
            pg[("postgres")]
            vol[("aiops-data volume — /data<br/>skills · secrets ·<br/>lakehouse: /data/lakehouse<br/>(per-tenant DuckDB files)")]
        end
        subgraph estate["profile: estate (demo target)"]
            prom["prometheus"]
            amgr["alertmanager"]
            grafana["grafana"]
            shop["demo-shop + fault inject"]
        end
        subgraph sso["overlay: docker-compose.keycloak.yml (optional)"]
            kcd["keycloak<br/>(demo realm aiops + provisioned<br/>aiops-* tenant realms;<br/>issuer host kubernetes.docker.internal)"]
        end
    end
    ext["AIOPS_MODEL / AIOPS_API_BASE<br/>(external LLM — or built-in mock)"]

    amgr -- "webhook" --> app
    app -- "PromQL / dashboards" --> prom
    app -- "PromQL" --> prom
    app --- pg
    app --- vol
    app -- "OIDC login (optional)" --> kcd
    app -- "admin API — realm provisioning" --> kcd
    app -- "completion" --> ext
    shop --> prom
    prom --> amgr
```

Production seam: the in-process detached-task + semaphore model (`runtime.py`) is where a real task queue/worker slots in — one function, no schema change. One control-plane replica by default; leader election (`leader.py`, a Postgres advisory lock) keeps the scheduler, sealer, sweeps and provisioner on one elected replica when more are run, and multi-replica is blocked today by the ReadWriteOnce `/data` lake volume and per-pod in-memory state (rate limits, ingest gate, SSE, and the in-process locks behind IdP domain claims, compliance landing and harvest recording) — see `availability.md`.

## 12. Endpoint → data map (selected) {#12-endpoint-data-map-selected}

| Flow | Endpoint | Writes |
|---|---|---|
| Ingest | `POST /webhook/{source}/{tenant_slug}/{token}` (token also accepted as a header) | `Alert`, `Incident`, `Investigation`, `AuditEvent` |
| Live view | `GET /api/aiops/{t}/investigations/{id}/stream` | — (SSE replay + live) |
| Decide | `POST /api/aiops/{t}/investigations/{id}/decisions` | `ApprovalRequest`, resumes engine |
| Remediate | `POST /api/aiops/{t}/incidents/{id}/remediate` | route exec → `Incident.status`, artifact |
| Escalate | `POST /api/aiops/{t}/incidents/{id}/escalate` | `Incident` (→ escalated) |
| Feedback | `POST /api/aiops/{t}/investigations/{id}/feedback` (`feedback.submit`; rating and/or 1-10 score; **UPSERT** per (investigation, user) — DB-unique) | `Feedback` (→ `SkillProposal` draft on first rating / new down-correction) |
| Model tiers | `GET\|PATCH /api/aiops/{t}/model-tiers` (GET any member; PATCH `pillars.manage`, merge-patch — `""` clears a tier) | `Tenant.settings.model_tiers`, `AuditEvent models.tiers_updated` (the generic tenant PATCH rejects this key) |
| Tier suggest | `GET /api/aiops/{t}/model-tiers/suggest` (`pillars.manage`) | — (deterministic suggestions + evidence; HITL — saving via the PATCH is the audited apply) |
| Model discovery | `GET /api/aiops/{t}/models` (`pillars.manage`) | — (gateway `/models` passthrough; OpenRouter pricing included when present) |
| Skill promote | `POST /api/aiops/{t}/proposals/{id}/approve\|retire` | `SkillProposal`, skills dir |
| Members | `POST\|DELETE /api/aiops/{t}/members...` | `TenantMember`, `AuditEvent` |
| Roles | `POST\|PATCH\|DELETE /api/aiops/{t}/roles...` | `TenantRoleDef` |
| Support access | `POST\|DELETE /api/aiops/{t}/support-access` | `TenantMember(support)`, `AuditEvent` |
| SSO discover | `POST /oidc/discover` (unauthenticated by necessity — reveals only found + provider name) | — |
| SSO login | `GET /oidc/login?idp=` → `GET /oidc/callback` (unauthenticated; idp rides the signed state cookie) | `User` (link/create, invariant-guarded), `TenantMember` (auto-provision; `members.manage` roles refused), `AuthSession`, `AuditEvent member.auto_provisioned` |
| Tenant IdP | `GET\|PUT\|DELETE /api/aiops/{t}/idp` (`members.manage`; PUT validates SSO-assignable roles + reserved/globally-unique domains under a claim lock) | `TenantIdP`, secret store (client secret write-only under `tenant-idp-{t}`), `AuditEvent idp.updated/idp.removed` |
| IdP test | `POST /api/aiops/{t}/idp/test` (`members.manage`) | — (issuer discovery fetch; SSRF-guarded, never 500s) |
| Keycloak provision | `POST /api/aiops/{t}/idp/provision-keycloak` (superadmin; 503 unless `AIOPS_KEYCLOAK_ADMIN_URL`; 409 if an IdP exists) | Keycloak realm `aiops-<slug>` + `TenantIdP` + secret store; realm + secret rolled back if the DB write fails; `AuditEvent idp.provisioned` (never the secrets) |
| Datasource | `POST /api/aiops/{t}/datasources` (+ `/test`) | `TenantDatasource`, secret store |
| Knowledge | `POST /api/aiops/{t}/knowledge` | `KnowledgeItem` |
| Benchmark | `POST /api/aiops/{t}/benchmarks/{builtin\|golden\|import}` | `BenchmarkRun` |
| Legacy feature grant/revoke | `POST\|DELETE /api/aiops/{t}/features/{f}` (superadmin) | Deprecated authenticated no-ops; historical rows are retained and availability does not change |
| Legacy terms accept | `POST /api/aiops/{t}/features/{f}/accept` (`members.manage`) | Deprecated authenticated no-op; features are included and no agreement or collection spec is created |
| Lake overview | `GET /api/aiops/{t}/lake` (`datasources.view`) | — (stats, catalog, retention, PII totals, specs) |
| Lake specs | `GET\|POST\|PATCH\|DELETE /api/aiops/{t}/lake/specs...` (+ `/seed-defaults`) (`datasources.manage`) | `CollectionSpec`, `AuditEvent` |
| Lake query | `POST /api/aiops/{t}/lake/query` (`datasources.view`) | — (single SELECT/WITH, read-only connection) |
| Lake erase / drop | `POST /api/aiops/{t}/lake/erase` · `DELETE /api/aiops/{t}/lake` (`members.manage`) | DuckDB rows / whole lake dir; `AuditEvent` (needle stored as hash only) |
| Chat ask | `POST /api/aiops/{t}/chat` (`chat.ask`, 202 detached) | `ChatSession`, `Investigation(trigger=chat)`, `AuditEvent chat.ask` (question hash) |
| Chat sessions | `GET /api/aiops/{t}/chat` · `GET\|DELETE /api/aiops/{t}/chat/{sid}` (`chat.ask`, own-user only) | delete removes the `ChatSession` |
| Chat live | `GET /api/aiops/{t}/chat/{sid}/stream` (`chat.ask`, own-user only) | — (SSE snapshot + live turn events) |
| X-ray launch | `POST /api/aiops/{t}/xray` (`datasources.manage` + active `xray` feature; budget finite + ≤ `aiops_xray_max_cost_usd`) | `XrayScan(running)`, `AuditEvent xray.launched`; detached scan task |
| X-ray list / detail | `GET /api/aiops/{t}/xray` · `GET /api/aiops/{t}/xray/{id}` (`datasources.view` + feature) | — (detail = live poll target: per-phase tokens/cost + totals + report) |
| X-ray apply | `POST /api/aiops/{t}/xray/{id}/apply` (per kind: `datasources.manage` or `knowledge.manage`) | DRAFTS only — disabled `CollectionSpec`/`RemediationRoute`, inactive `[x-ray]` `KnowledgeItem`; `AuditEvent xray.applied` |
| X-ray cancel | `POST /api/aiops/{t}/xray/{id}/cancel` (`datasources.manage`) | `XrayScan` → cancelled (immediate; task bails at next phase boundary), `AuditEvent` |
| X-ray delete | `DELETE /api/aiops/{t}/xray/{id}` (`datasources.manage`; 409 while running) | removes the `XrayScan` row (report Artifact survives), `AuditEvent` |
| Artifacts | `GET /api/aiops/{t}/artifacts[?source_type=&source_id=&kind=]` · `GET .../artifacts/{id}` (`incidents.view`; `source_type=xray` rows also need `datasources.view` + active `xray` feature) | — (append-only; written only via `create_artifact`) |
| RCA export | `POST /api/aiops/{t}/incidents/{id}/rca-export` (`incidents.act`; idempotent per investigation) | `Artifact(kind=export)`, `AuditEvent artifact.created` |
| Knowledge activate | `POST /api/aiops/{t}/knowledge/{id}/activate` (`knowledge.manage`) | `KnowledgeItem.active=true` (supersedes active same-name), `AuditEvent knowledge.activate` |

MCP surfaces mirror these through the canonical REST handlers: `/mcp/aiops` (tenant permissions plus the PAT scope ceiling) and `/mcp/admin` (superadmin, with admin scope required for every PAT-authenticated tool). Both accept a PAT (`ntp_`) or OIDC bearer. Workspace-bound URLs select one tenant; they do not turn the PAT into a workspace-restricted credential. The [operation reference](/help/mcp-operations.md) documents the current names, aliases, nested request bodies and asynchronous results.

## 13. Governance layer (change hub · risk register · tamper-evident audit · certification) {#13-governance-layer-change-hub-risk-register-tamper-evident-}

Added for the enterprise acceptance scope (Aug-2026). Four subsystems close the loop between "the agent proposed something" and "an auditor can verify what happened":

```mermaid
flowchart TD
  subgraph hub["Change hub — change.py"]
    RR[run_and_record<br/>the ONE execute+record chokepoint] --> CR[ChangeRecord<br/>minted BEFORE the write]
    CR --> AZ{authorize<br/>window + tier quorum}
    AZ -- unmet L3 --> PA[pending_approval]
    AZ -- ok --> EX[execute route / gitops PR]
    EX --> LKG[pre-change state capture<br/>pre-image / snapshot / method=none]
    EX -- failed --> RB[rollback_sweep_tick<br/>auto restore-PR ≤L2, else escalate]
  end
  subgraph risk["Risk register — risk.py"]
    CF[CloudFinding upsert<br/>ASFF / Defender, scrubbed] --> RD["RiskDecision<br/>accept = Suppress / avoid = Fix / mitigate (no new ones) — HUMAN, append-only"]
    RD -- accept --> SUP[suppress re-sends until expiry]
    RD -- avoid/mitigate --> CR
  end
  subgraph audit["Audit chain — audit_chain.py"]
    AE[AuditEvent rows] --> SEAL[async sealer<br/>per-tenant sha256 chain, seq + prev_hash]
    SEAL --> VER[GET /audit/verify<br/>replay by seq]
  end
  subgraph cert["Certification — benchmark.py"]
    BR[BenchmarkRun full catalogue] --> CERT[certification manifest<br/>bundle_sha, chain-anchored]
    CERT --> VER
  end
```

Load-bearing decisions: the ChangeRecord exists *before* the write so a crash still leaves a ledger row; rollback drafts from agents are never auto-executed; risk decisions are append-only with mandatory statements (supersede, never edit); the audit chain is sealed asynchronously by the single-writer scheduler (≤60s unsealed window is the accepted trade); a certification is a re-verifiable bundle (`certification.json`) whose sha is itself an audited, chain-anchored event — the pass-rate claim can be replayed, not just quoted.

Posture over time comes from the same register. The leader's scheduler tick first saves one `PostureSnapshot` per workspace for the UTC day that just closed (`risk.snapshot_tick`, insert-only; earlier days, and a day saved more than an hour after its close, are rebuilt from the records and marked reconstructed). **Posture trend** on Compliance findings (`GET /api/aiops/{t}/findings/score`, by day, ISO week or month, sliced by account and source) reads saved days as they were saved and rebuilds only the days not saved yet.

## 14. Execution plane v2 (durable claims · per-tenant runners · managed test lab) {#14-execution-plane-v2-durable-claims-per-tenant-runners-mana}

```mermaid
flowchart LR
  subgraph Control plane — one replica by default, leader-elected singleton loops
    API[FastAPI + scheduler + ingest]
    Q[(Investigation / XrayScan / FactoryRun rows<br/>ARE the queue: runner_id, lease_expires_at, attempt)]
    API --> Q
  end
  subgraph Runner pod per tenant
    CL[claim loop<br/>atomic conditional UPDATE] --> EN[engine run<br/>per-step transcript commits]
    EN --> HB[heartbeat lease]
  end
  Q <--> CL
  EN -- crash --> RES[lease expiry → attempt+1<br/>resume from transcript]
  API -- AIOPS_EXEC_MODE=remote --> NOP[CP only enqueues<br/>deploys never kill db-mode jobs]
  subgraph Managed test lab — testfactory.py
    PROV[provision estate<br/>write-role kubectl ds, agent-invisible] --> INJ[inject → grade → heal]
    INJ --> PART[per-case rows persisted<br/>AS THEY GRADE — result.per_case_partial]
    INJ --> SETTLE[settle gate:<br/>pods green AND alerts silent]
  end
  Q --> PROV
```

The factory runs the same claim/lease machinery as investigations; a lab run interrupted by a control-plane restart is failed-and-healed (never resumed), but its graded rows survive incrementally (`per_case_partial`) so the work is not lost. Two lab-specific validity mechanisms are architectural, not incidental: the **settle gate** (a lane is released only when the shard's pods are Ready *and* its alerts have drained — pods-green alone let the previous fault's late alerts poison the next case) and **stale-run escalation suppression** in the engine (a finished investigation must not reopen an incident that resolved while it ran; the reopen edge belongs to the post-close verify). Both were measured into existence by the acceptance loop — see `tasks/overnight-journal.md` rounds 8–11 for the evidence.

## 15. Correlation — how a burst of alerts becomes ONE incident {#15-correlation-how-a-burst-of-alerts-becomes-one-incident}

The single most load-bearing piece of product logic (`app/aiops/ingest.py` + `correlate.py`): every noisy estate produces hundreds of alerts per fault, and everything downstream — one investigation, one approval trail, one RCA — depends on folding them into exactly one incident without ever merging two different problems.

```mermaid
flowchart TD
  W[Webhook alert arrives<br/>alertmanager / grafana / datadog / cloudwatch<br/>servicenow / jira / securityhub / defender ...] --> N[Adapter normalizes to CANONICAL labels<br/>service, component, severity + tenant label_map]
  N --> D{Dedupe: same fingerprint<br/>still active on a joinable incident?}
  D -- yes --> T[refresh incident correlation clock<br/>suppressed, no new row]
  D -- no --> K[correlation key = service label<br/>fallback fingerprint]
  K --> J{Open incident<br/>with same key?}
  J -- yes --> JOIN[JOIN it: severity may UPGRADE,<br/>never downgrade; L3 fields re-derive]
  J -- no --> F{Label-correlation fold:<br/>another service, same CAUSE<br/>within window?}
  F -- hit --> JOIN2[fold into that incident<br/>one cause, many services]
  F -- miss --> OPEN[OPEN new incident<br/>severity = max of alerts<br/>criticality = declared max<br/>priority = severity x blast radius,<br/>+1 step if mission-critical<br/>baseline harvested for post-fix verify<br/>suspect change pinned on event time]
  OPEN --> G{Dependents of this service<br/>have open incidents<br/>inside the grouping horizon?}
  G -- yes --> REROOT[merge them in: the top-most<br/>failing service is the root]
  OPEN --> H{Depends on a service<br/>over a foldable edge,<br/>not a P1?}
  H -- yes --> WAIT[analysis waits for the grouping wait:<br/>that service's alert takes it in]
```

Decisions worth defending to an auditor:

- **The key is the `service` label**, first and always. Everything the tenant maps into `service` via `label_map` becomes the folding identity. Tickets (ServiceNow/Jira/ADO) are normalized the same way — `cmdb_ci` / component becomes `service` — so a ticket *joins the live telemetry incident* instead of opening a parallel one, and the incident adopts the originating ticket (no ticket → incident → ticket loop).
- **No recency window on the join.** An open incident absorbs same-key alerts *however old it is* — a recency window split one nine-hour problem into siblings. Status is the separator: `resolved` is the only closed state.
- **Joinable = open, investigating, awaiting_approval, remediating, and escalated.** Escalated is included on purpose: while a human works the fix, a re-fire must fold into that incident, not open a duplicate next to it. The dual of this rule: a *finished* run may not push a resolved incident back to escalated (stale-run suppression, §14) — otherwise a dead investigation could silently re-arm the join target for the next, unrelated fault.
- **Grouping follows the service graph, never a count (0.1.147).** A fault that cascades across services (DB down → five callers erroring) becomes one incident on the top-most failing service: a caller arriving after its cause folds into it (rule 1 below), and a cause arriving after its callers opens its incident and takes theirs in (`grouping.group_on_arrival`, audited `incident.grouped`, the merged row keeps `merged_into_id`), both through the same fold gates and inside the grouping horizon (`aiops_storm_group_window_s` + `aiops_storm_spread_window_s`, 900 s). A caller's analysis waits (the grouping wait, 0.1.148: `aiops_grouping_wait_s` 300 s, `grouping.awaited_roots`, never for a P1, never for an alert none of whose episodes was analysed), so a cause arriving after a caller's analysis would have escalated still takes it in — an escalated incident is never regrouped; `grouping.release_waits` starts the analysis at the wait's end. A reopen takes rule 1 first, so a caller whose root is open joins it instead of reopening beside it. The grouping agent (`grouping.group_triage_tick`, one bounded turn per related group) covers the edges that may not fold. A burst of unrelated services opens one incident each (the shared-alert-name fold, 3 services, is the one fold without a relationship); the 15-per-120 s storm collar and its `__storm__` incident are gone, and a blob an earlier release left open is no longer split.
- **A dependency folds only when the workspace trusts its source.** The same fold step (`correlate.find_fold_target`) joins an alert to an open incident whose blast radius holds its service — by default through **declared** edges only, which a member with `datasources.manage` adds as `A → B` (A depends on B) on Operate → Service correlation → Service dependencies (0.1.110) or through `POST /api/aiops/{t}/topology`. Other sources fold only when listed in `aiops_topology_fold_origins`, and then only for a direct caller of a dependency with at most 10 direct callers, over an edge seen within `aiops_topology_traced_ttl_h` (168 h by default).
- **Severity only upgrades on join; the worst correlated signal defines the incident.** Declared criticality (`low` < `medium` < `high` < `mission-critical`, an alert label or the tenant `criticality_map`, never inferred) only rises the same way. Priority (P1–P4) derives from severity (critical P1, error P2, warning P3, else P4), one step higher when the blast radius (topology reverse-BFS, 3 hops) holds two or more services and one more for a mission-critical resource, floor P1 — re-derived on every join and merge (`ingest.reprioritize`). Numeric alert values are harvested as the incident's `baseline` — that is what the post-fix verify compares against.
- **Resolution is symmetric**: when every correlated alert has reported resolved, the *condition* has ended — the incident auto-closes only from `open`/`investigating`. An incident a human holds (awaiting approval, remediating, escalated) is never yanked closed by telemetry. The one exception is an escalated incident whose own adopted ticket — the customer's, sent in through an inbound ticket webhook — is resolved at its source with nothing else firing: that person closed it, in their ticket system (`incident.resolved` by `ticket-resolved`), unless its fix still waits for a decision under the US01-AC1 hold switch.

The test-lab acceptance loop measured this machinery under load: the lab tenant runs 12 estates concurrently, and rounds 8–11 of `tasks/overnight-journal.md` document the two bugs found and fixed at the correlation boundary (late-notification incidents born after quiet, and the stale-run escalation that made a closed key joinable again).


## Next steps

- [All Reference pages](/help/reference/llms.txt)
- [Documentation home](/help/)