Release notes 0.1.166

29 October 2026: The scale release: webhooks that no longer drain the database pool under load, a change capture that owes what it could not read instead of dropping it, and an estate history that…

Release notes · Updated 2026-10-29 · Describes release 0.1.173

0.1.166 — 29 October 2026

The scale release: webhooks that no longer drain the database pool under load, a change capture that owes what it could not read instead of dropping it, and an estate history that stops growing. It also clears the findings of an AWS account you disconnected and fixes one line of incident copy. Schema, additive and created at start: the table loginfailure. The chart adds a dbPool block with defaults and sets AIOPS_API_REPLICAS; no values change. New audit action compliance.account_disconnected. One default changes behaviour and cannot be undone: the estate history is thinned (opt-out below). Give every Amazon SNS HTTPS subscription a delivery policy before or with the upgrade (below).

Webhooks under load

A busy workspace answers 503 instead of failing. Five workspaces sending about five webhooks a second drained the control plane's database pool: deliveries failed with 500 after 30 seconds and were lost. A delivery now authenticates, passes its rate limit and waits for its workspace's turn without a database connection. On the four routes whose senders retry a 5xx on their own — /webhook/alertmanager, /webhook/cloudwatch-sns, /webhook/azure-monitor and /webhook/securityhub — it waits at most AIOPS_INGEST_LOCK_TIMEOUT_S (5 seconds); then the platform answers 503 with Retry-After: 5 and stores nothing. The retry is deduplicated like any re-send. Every other route (Grafana, Datadog, Dynatrace, the ticket and change feeds, /webhook/generic) waits as long as it takes, as before, and gets a 503 only when no database connection frees within 30 seconds.
Retry advice for Amazon SNS. Prometheus Alertmanager, Azure Monitor action groups and EventBridge API destinations retry a 503 for long enough. Amazon SNS retries an HTTPS delivery only 3 times, 20 seconds apart, unless the subscription has a longer delivery policy: give every SNS subscription that delivers CloudWatch alarms or Security Hub findings to the platform one (step 3 of Set up alert sources has the command), or a delivery that keeps meeting a busy workspace for a minute is lost. A sender of your own should retry a 503 with the same payload after the Retry-After delay.
Back-pressure per sender. Each workspace's rate limit counts per replica (the limit divided by AIOPS_API_REPLICAS), and a payload larger than a replica's share passes on a full bucket instead of being refused for ever. A 503 hands its rate charge back, so the sender's retries do not run into a 429.
A larger pool, sized from the chart: each control-plane process opens up to 10 + 20 overflow connections (was 5 + 10), from the chart's new dbPool block. A webhook process uses at most half of them at once (AIOPS_INGEST_ADMISSION); in embedded mode, runs claiming and building their prompts use at most a quarter (AIOPS_EMBEDDED_RUN_SLOTS).
A disabled workspace or a rotated token is still refused at once. The workspace's slug, token hash and settings are cached for AIOPS_TENANT_CACHE_S (30 seconds), and the platform reads the workspace again under its lock before it stores anything. A token minted or a workspace enabled on another replica is accepted on its first delivery.
Metrics: aiops_ingest_busy_total{reason} counts each 503 (turn, admission, lock, pool). The example alert rules stop counting a 503 with Retry-After as an ingest failure and add AiopsIngestBusy.

No lost change on a late CloudTrail delivery

Owed change windows. A scheduled change pull reads at most 500 kept events (1,500 read) per connection; what did not fit used to be dropped. Now the connection owes the unread part of its window and reads it on the next pulls, oldest first, each region, EKS cluster or subscription taking its own share. The change.harvested audit row says what is owed and why, and aiops_change_harvest_owed_seconds{tenant} shows how far behind the oldest unread second is (alert AiopsChangeCaptureBehind above an hour).
Late deliveries are read. CloudTrail and the Activity Log deliver some events up to 15 and 20 minutes late, so every span is read once more after 20 minutes. A long harvest interval (a daily one) still reads that late span on every pull. A throttled page is asked again after a backoff instead of failing the pull.
Gaps are counted, never silent. A connection that comes back after more than a day without a scheduled pull reads its last day and counts the span before it lost; so does a window older than the 90 days the sources keep.

Estate history

The estate history stops growing — on by default, not reversible. Each save of an estate item keeps the newest version, the active one and every version of the last 48 hours, then the first and the last of each day, and removes versions older than 30 days, at most 100 a save. A harvest keeps the newest 7 Turtle ABox snapshots per connection (never an x-ray scan's, one an attestation cites or a certification). Past 48 hours, last-known-good answers to the day rather than to the version. Nothing is thinned while record retention archives (or is paused), or while the retention sweep is switched off (AIOPS_RETENTION_SWEEP_ENABLED=false): thinning runs only when the sweep is on and its action is delete. A removed version comes back only from a database backup. To keep everything, set AIOPS_ESTATE_HISTORY_DAYS=0 and AIOPS_ESTATE_ABOX_KEEP=0 through extraEnv in the same upgrade.
Re-harvest after captured changes at most once an hour per connection (was every minute); until then the touched resources stay marked stale and last-known-good reads them live.

Background work

Detached pulls. The change harvest, the scanner pull, the estate inventory and the synthetic-check suggestions run beside the scheduler tick, one at a time, so a pull that catches up no longer holds the tick back. The estate inventory takes that turn per connection, so a daily sweep over many connections lets the change harvest and the scanner pull run between two of them. A scanner pull lands its findings 200 at a time (AIOPS_SCANNER_CHUNK), so a webhook waits behind one chunk, not the whole pull. A pull stops between chunks when its replica is no longer the leader, and the expiry sweep of risk decisions waits for the workspace's ingest.
Login failures in the database. The login lockout counts failures in the new loginfailure table (a hash of the email and the time, never the address), so it trips once whichever replica got the attempts and a restart keeps the count. Rows older than a day are removed. A lock in place before the upgrade does not carry over.
State that holds across replicas. The SSO email-domain claim, the MCP connection receipt, a full audit-chain replay and the SMTP and guardrails saves are serialised in the database. The control plane still runs one replica.
A scanner re-stamp no longer rewrites the finding: a re-sent finding whose only change is its update or observation time keeps its stored payload.

Compliance and incidents

Findings of a disconnected account clear. After a scanner pull whose every AWS read was whole, an open Security Hub finding of an account that a read connection once named (its connection disabled, or deleted) and no enabled one names any more, that the pull did not return and that was not seen for 48 hours is cleared as account no longer connected — never counted as fixed (audit compliance.account_disconnected). Before, it stayed open until Security Hub's 90-day retention. Nothing is cleared while an enabled AWS connection's account is unknown, and an account no connection ever named (its findings come by webhook) keeps its findings until that retention, as before.
The incident's plain line names an action. An incident whose plan has no runnable step read You: Not applicable — no runnable fix.; it now reads Next: run the manual steps yourself, then mark the incident resolved.

Review fixes

The rule dictionary the automation in your account reads re-checks the workspace first: a workspace disabled since gets nothing.
The other fixes from the release review are part of the items above: the daily-interval late span, the leader check on detached pulls, the first-delivery acceptance of a new token, no thinning while retention archives or its sweep is off, the rate charge handed back on a 503 (the change feed's too), the change feed re-reading the workspace before it records a second automation, and a webhook-fed account's findings kept open.