Security and data handling

Understand data boundaries, credential storage, redaction and approval rules.

Concept · Updated 2026-10-27 · Describes release 0.1.164

Who this chapter is for

Your security reviewer, and anyone who has to sign off on connecting this platform to production systems. Everything below describes shipped behavior, with the file-level mechanism named where it helps you verify it. Where the platform is weaker than you would like, this chapter says so rather than leaving your penetration tester to find it.

Three claims are load-bearing, and the rest of this chapter is the evidence for them:

1. Credentials are write-only. They enter the platform, they are used to make a call, and no interface reads them back.
2. No write reaches your estate without a named human — either a person releases it, or, only where your workspace owner turned on automatic API fixes, it repeats unchanged a fix a named approver certified and a named person released before, which the platform verified. The agent cannot loosen that itself, and neither can your own operators. Ticket routes, which run when an incident opens or resolves, write only to a ticket, paging or webhook system, never through a write connection; a hand-written one runs only once an approver certified it (Runbooks and automatic fixes).
3. Platform staff are not members of your tenant. They see your tenant's data only through a membership row you can see and revoke.

Where every credential lives

The tenant secret store

Every credential you type into the platform lands in one place: a per-tenant secret store, backed by a dedicated database table (app/aiops/secrets.py). The record that references it — the datasource row — holds only a secret_ref pointer, never a value.

Four properties of the store matter to a reviewer:

• Encryption is conditional. When the platform operator sets AIOPS_TENANT_SECRET_KEY, the stored JSON is encrypted with Fernet under a SHA-256-derived key. When that key is not set, values are stored as plaintext JSON and the application logs a loud warning at every startup: "tenant datasource secrets are stored in PLAINTEXT in the database." Ask your platform operator to confirm the key is set on the instance serving you. This is a yes/no question with a yes/no answer.
• Turning encryption on is retroactive. A plaintext row read while a key is configured is re-encrypted in place on that read.
• It fails closed, not open. An encrypted row read with the key missing or wrong returns nothing and logs an error. The tool call then fails for want of a credential rather than silently running unauthenticated.
• Nothing outside that module touches the table. The store is the designed replacement seam for a Kubernetes Secret, Vault or OpenBao backend.

The limit, stated plainly: encryption is at rest only, and the key is an environment variable in the same process environment as the application. Anyone who can read that process's environment can decrypt the store. External key management is a designed seam, not a shipped feature.

What the console can and cannot show back

• Secret fields render as password inputs and are write-only. No REST endpoint, MCP tool or console screen returns a stored secret value.
• On Edit, a stored secret is never filled in: a single-line secret field shows a masked placeholder and the label (secret, write-only — blank keeps current), a multi-line one such as a kubeconfig reads (unchanged — paste full content to rotate), and the dialog states: "Stored secrets remain unchanged unless you explicitly replace them or switch to keyless." An AWS or Azure connection shows no secret field while Authentication method is Keep current authentication (Stored credentials remain unchanged.); choose Replace access keys or Replace a client secret to enter a new value, which is stored write-only.
• The Company IdP panel shows a secret stored badge and nothing else.
• Deleting the datasource deletes its secrets, and only after the database row commits — a failed delete never destroys a credential.

The caveat you must design around: only fields the connector marks as secret get this treatment. Non-secret configuration — base URLs, regions, account ids, the operator notes in Description — is returned by the datasources API and readable by anyone holding datasources.view. Never paste a credential into a non-secret field.

Rotate a credential

Permission required: datasources.manage.

1. Open Data sources.
2. Find the datasource and select Edit.
3. Leave every field you are not rotating blank.
4. Paste the new value into the secret field, in full — the platform validates YAML, JSON and PEM formats at save time, so a kubeconfig that lost its line breaks in the paste is rejected with a 400 rather than failing mid-investigation.
5. For investigation/shared access, select Connect and verify and inspect the read-check result. Opted-in AWS/Azure execution-only access offers Save and check access; other execution credentials are saved without that check. A read check never proves permission for every write.

A datasource.update row in Audit records the changed field names, never secret values. The rotation is written to the secret store only after the database row commits, so a rejected save never leaves a rotated credential behind.

What leaves your network, and to whom

There are seven categories of outbound traffic.

1. The model gateway

The investigation loop calls one language model endpoint, configured by your platform operator as a model id, a base URL and an API key. If your operator points that base URL at a model you host, nothing in this category leaves your network at all. If no model is configured, the platform runs a deterministic mock and makes no model call whatsoever.

What a request to that endpoint contains:

The honest part: tool output is the substantial thing that leaves, and by default it is sent raw. The optional tool-output PII scrub (aiops_pii_scrub_tool_output) ships off. Estate x-ray is the exception — it forces the scrub on, with no flag to disable it, because estate data flows heavily there. If your threat model requires that no unredacted log line reaches the gateway, ask your platform operator to enable that scrub; see PII: the floors and the switches below.

Two mechanical limits bound the volume: any single tool output is truncated at 24,000 characters before it enters the conversation, and once total tool output in one run exceeds roughly 200,000 characters the oldest tool messages are elided from the context.

2. Web search

The web_search tool calls one fixed search endpoint — DuckDuckGo's HTML endpoint by default, or Tavily or Brave when your operator configures an API key. Only the query string leaves, truncated to 256 characters, and the query passes the PII scrubber before egress. That is deliberate: the query is the one field a hijacked agent could use to smuggle estate data to a third party, so credential and identifier patterns are neutralized on the way out. web_search is never approval-gated, because there is nothing sensitive left in it to gate.

3. Web fetch and arbitrary HTTP

Both honor the SSRF guard, which resolves the target host and rejects private, loopback, link-local and cloud-metadata addresses. Your own configured datasource hosts stay reachable.

4. Your own datasources, and the clouds you connect

The agent calls the endpoints you configured, with the credentials you supplied. For the native cloud connectors this means fixed vendor endpoints, and only for a cloud you actually connected: AWS STS and service endpoints, Microsoft's identity and management endpoints plus Log Analytics, Google's OAuth and service endpoints. For a GitOps remediation the platform calls your source-control API — hosted or self-hosted, whichever base URL you configured. When an Amazon SNS subscription points at the intake, the platform confirms it by fetching the SubscribeURL SNS sent, and only a genuine https://sns.<region>.amazonaws.com address.

5. Your identity provider

If SSO is enabled, the platform fetches the issuer's discovery document and JWKS to verify token signatures, and caches both for one hour. Each fetch must finish within 10 seconds and stay under 256 KiB, or the token is checked against the issuer's userinfo endpoint instead. The platform asks for uncompressed answers and refuses a compressed one. A signing key larger than 4096 bits, or with a public exponent that is not an odd number from 3 to 65537, is refused.

6. Notifications and pagers

The targets your workspace owner configures under Settings · Account → Notifications & on-call: the Teams, Slack and webhook URLs they enter and the SMTP relay that sends email. A pager goes only to its vendor's fixed endpoint — PagerDuty events.pagerduty.com (EU events.eu.pagerduty.com) or Opsgenie api.opsgenie.com (EU api.eu.opsgenie.com) — never to a URL typed in. Every message passes the PII scrubber before it leaves.

7. The price lists

Once a day by default the control plane downloads published model prices from two fixed addresses: the AWS price list (pricing.us-east-1.amazonaws.com) and models.dev (models.dev/api.json). It sends nothing about your workspace. An installation without that egress turns the sync off and records prices by hand.

What never leaves

There is no telemetry, no analytics beacon, no usage reporting and no phone-home to the platform vendor. The console is a self-contained bundle: no CDN script, no external font, no third-party tag. You can verify this at the network layer with an egress allowlist containing only the seven categories above — the platform will function normally.

PII: the floors and the switches

The scrubber replaces detected values with a correlation-preserving pseudonym:

[REDACTED:email:9f4c2ab71e08d5c3]

The hash is the first 16 hex characters of SHA-256(tenant_salt + value). The same value produces the same token within your tenant, so joins and correlation survive redaction; a different tenant salt makes the same value produce a different token, so tokens are not linkable across tenants. The transformation is one-way. A scrubbed record cannot be un-scrubbed, which is why the optional patterns are deliberately conservative — a false positive destroys data permanently.

Chokepoints with no off switch

These are code, not settings. No endpoint, flag or role can disable them.

The always-on detectors are: secrets (API keys, tokens, private-key blocks, JWTs), email addresses, payment card numbers (Luhn-validated), and phone numbers. They are listed read-only on the Guardrails page so you can see the baseline.

Chokepoints you configure

Say this out loud in your review: the last three are install-wide environment settings, not per-tenant controls. You cannot turn tool-output scrubbing on for your tenant alone from the console. Raise it with your platform operator as a change to the deployment.

You may add up to 20 regular expressions. Each is compiled and validated on save — an invalid expression is rejected with a 400, and a pattern that later fails to compile is skipped rather than taking ingestion down. The Guardrails page flags how many recommended patterns you have not yet adopted, which is the review signal when new defaults ship.

A detection is surfaced, never silently swallowed

When the scheduled collector finds PII in telemetry it is pulling from your systems, it writes a pii.detected audit row and opens a compliance incident against the datasource. Your own leak is reported to you, not quietly redacted away.

Erase a value from the lakehouse

Permission required: members.manage. Requires the lakehouse feature.

1. Open Lakehouse.
2. Enter the value to erase — at least 3 characters.
3. Confirm the erasure.

You should see a deleted-row count returned, and a lake.erased audit row whose payload carries a hash of your search term and the count, never the term itself. The erase matches both the literal value and the value's own pseudonym token, so erasing an address also removes the rows the scrubber already protected. Lakehouse also offers a full drop of the tenant's lake.

The limits: artifacts are append-only and have no erase path by value. Investigation transcripts have no per-value erase path. Closed records — resolved incidents, finished scans, terminal changes, cleared findings, with their runs and artifacts — age out on the workspace's retention window (Guardrails & cost → Data retention, default one year); the audit trail never does. There is no delete-tenant endpoint in this release — removing a tenant's data entirely is a platform-operator database task.

The record archive. When the platform operator configures an archive bucket, records past the window are archived before they are removed: one compressed file per closed month of a workspace in a dedicated S3 bucket of the platform's account, encrypted with a customer-managed KMS key, with credentials and secrets masked when the file is written (PII pseudonyms already stored stay pseudonyms; nothing new is pseudonymised). The bucket's Object Lock (governance mode) keeps every file for the period the operator set — 5 years by default — and only the control plane's own role can read or write it; the platform holds no permission to delete or shorten it. Chat sessions and estate x-ray attachments are never archived, and removals a person asks for (an x-ray scan or attachment, a trashed test dataset, a knowledge item) stay immediate. Erasing a record from the archive inside its retention period is a platform-operator break-glass procedure, not a console action: ask your operator. An archived incident or change stays readable from its link and from the Archive page, read-only, with the same permission as the live record (GET /{workspace}/archive/{incident|change}/{id}); a per-month fine-tuning export and back-test cases (members.manage) are built through the control plane, and the export runs the full PII scrub with the workspace's salt before a byte leaves the platform.

The write gate as a control matrix

Investigation datasource tools reject writes, regardless of whether the connected credential has provider write permissions. What counts as a write is decided by an intrinsic floor in code; your per-agent approval list can only add gates on top of it. The matrix per tool family, the guarantees that hold whatever the credential mode (write-role datasources invisible to the agent, shared credentials keeping the read boundary, chat and estate x-ray read-only), why lake_query is ungated, who may widen the gate and what untrusted-input fencing does and does not promise are on Read and write roles, and the write gate.

Tenant isolation and the staff data boundary

• There is no superadmin bypass. Every tenant-scoped read and write resolves permissions through one function that looks for a real membership row. Platform staff hold no implicit access to your incidents, investigations, chat sessions or lake.
• Unauthorized reads return 404, not 403. A non-member cannot learn that your tenant exists. A member missing a specific permission gets 403 — that distinction is intentional and is enforced on the REST API and the MCP servers alike.
• Expired memberships are absent, not stale. A membership past its expiry is treated as missing and the row is deleted on the spot.
• A disabled tenant fails closed on both the cookie session path and the SSO bearer path, at the same single choke point.
• Chat sessions are strictly own-user. Another member of your own tenant — superadmin included — gets a 404 on someone else's session. The one processor of every member's turns is a private network agent your owner created with Answers workspace chat; an agent token created without it never receives a chat turn.
• SSO can never mint an administrator. A group-to-role mapping that would grant members.manage is rejected when the mapping is written, again when the role is edited, and a third time at login.

The precise boundary, stated honestly: platform staff can see tenant configuration and cross-tenant aggregates — the tenant list, counts, connector catalog — through explicit staff-only code paths in the platform console. They cannot see your tenant's incident, investigation, chat or lake data without a membership row. That row is the boundary, and it is visible to you.

Support access is expiring, badged and revocable

When a platform engineer needs to see your data to help, they self-grant a support membership. What that means concretely:

The honest part: the grant is self-served. You are not asked first. You see it and you can end it, and it ends itself.

Revoke a support grant

Permission required: members.manage.

1. Open Members & roles.
2. Find the member carrying the support badge.
3. Select Remove.

You should see the member disappear from the list, and a support.revoked row in Audit naming who removed whom.

The audit trail as evidence

Audit rows are append-only and never updated. Each row carries a tenant, an actor (a user id, agent:<pillar>, or system), a dotted action, the id of the entity touched, a JSON payload and a timestamp. Reading your tenant's audit requires report.view. Filtering is server-side across action, actor, target and payload, and the filter chips are built from the action values your tenant has actually produced — so they cannot drift from what the platform writes.

What the rows that matter actually prove:

Gaps you should plan around. There is no built-in SIEM forwarder: you pull the trail. Export on the Audit page, or GET /api/aiops/{tenant_id}/audit/export on a schedule with a read-scoped personal access token, returns a period or the next page after a seq cursor (after_seq), with each row's seq, prev_hash and row_hash. scripts/verify-audit-export.py re-verifies an NDJSON file offline, with no access to the platform, before you ship it to your SIEM or archive. And audit rows live in the same database as everything else, chain head included — this is not a write-once store. Keep the exported files in your own write-once storage: that archive is the anchor that shows whether the online trail was rewritten after you took it. The steps are in Audit log → Export evidence for an auditor.

Tokens and session handling

Other properties worth recording in a review:

• A personal access token resolves to its owning user, so that user's RBAC applies unchanged; the token's scope (read / write / admin) can only cap it further, never widen it.
• Session cookies are set HttpOnly and SameSite=Lax, and Secure whenever the request arrives over HTTPS.
• Cross-origin mutations are rejected by an origin guard before authentication is even considered, which closes drive-by browser posts and DNS rebinding.
• Sign out performs a real form post so that, for an SSO user, the provider's end-session redirect fires and the next login cannot silently reuse a still-open provider session.
• Disabling a user account takes effect on the very next request, on both the cookie path and the token path.
• SSO bearer authentication requires the iss claim, matches it against the platform issuer or one of your enabled tenant IdPs, requires the token to name that issuer's client in aud or azp, verifies the RS256 signature against the issuer's JWKS, rejects opaque tokens outright when a client id is configured, and maps only onto an existing account — bearer authentication never auto-creates a user or a membership.

Changing a password, or a staff password reset, ends the account's other browser sessions; a self-service change keeps only the session that made it. Personal access tokens are separate credentials and are not affected.

Two gaps, named: a tenant owner cannot revoke another member's personal access token from the console — tokens are own-user only. And there is no separate "sign out everywhere" control. For both, the escape hatch is disabling the user account, which is a platform-staff action and takes effect immediately. If an employee leaves, disable the account; do not rely on token expiry.

Questions your security team will ask

Limits we will not soften

• Secret encryption is at rest only, keyed by an environment variable in the application's own process environment. External key management is a seam, not a feature.
• The tool-output and alert PII scrubs are install-wide and default off. Per-tenant control does not exist.
• Untrusted-input fencing bounds influence, not capability. The write gate is what actually contains the blast radius.
• The audit trail has no SIEM forwarder (exports are pulled, not pushed) and no write-once guarantee.
• Artifacts and investigation transcripts have no erasure path, and there is no delete-tenant endpoint.
• A tenant owner cannot revoke another member's personal access token, and there is no sign-out-everywhere.
• Minting a new webhook ingest token invalidates the old one with no overlap window.

Related reading: Manage connections and Read and write roles for the connection-level view of credentials and roles, Guardrails and cost limits for the approval floors and cost caps, Permissions for the full permission matrix and Members and roles for included feature availability, and REST API for token-authenticated API access.

Next steps

• All Administer a workspace pages
• Documentation home