Built-in detection (AWS)

CloudWatch rules over your estate inventory for accounts with no alarms — outages, and latency or faults above each service's own baseline; read-only, shadow first.

Concept · Updated 2026-10-28 · Describes release 0.1.164

Built-in detection watches the AWS resources in your estate inventory with a fixed set of CloudWatch rules, for accounts where nobody created alarms. It is off until a workspace owner turns it on, starts in shadow mode, and only reads: it lists metrics and reads their datapoints, and never creates an alarm, an anomaly detector or anything else in your account. No model decides anything here. A rule that fires sends an ordinary alert (source builtin) into the same pipeline as your own alerts: deduplication, correlation, grouping, the incident and, for a critical rule, an investigation.

Where it lives: Settings · Data in → Datasources (in the product: /portal/datasources) → Built-in detection (AWS).

What it watches

Thresholds and M-of-N come from AWS's Recommended alarms (Classic) list, the ALB and NLB metric pages and OpenSearch's recommended alarms. Ours marks a number no vendor source gives; every number and its reason is also in Thresholds and why. R26 to R30 (0.1.153) watch degradation as well as outages, where nobody created an alarm: a service answering slower or failing more than it normally does is an incident too. "Gate self" means the rule runs only where its own metric is published (a conditional metric); "gate resource" means it runs on every resource of its kind that published anything in the last 3 hours (its counts are published only when non-zero).

Each row on the card links to its source page. A rule AWS says not to apply is skipped and the card says why (for example "storage autoscaling on", "not a read replica", "retention not read").

Baselines (R26, R27, R28)

Latency and fault rates have no number that fits every service: 300 ms is normal for one API and an outage for another. These three rules compare each series with its own baseline: the 95th percentile of its 5-minute values over the last 7 days. A week holds the weekly cycle, and the 95th percentile lets one bad hour of it pass without hiding a regression. The threshold is the larger of two floors, so neither a fast service's jitter nor a slow service's normal swing alerts: for latency, 1.5 × the baseline and the baseline + 100 ms; for the fault rate, 5 % and the baseline + 5 points. AWS's own TargetResponseTime alarm says the threshold "depends on your situation" and to "analyze the historical behavior of this metric": the baseline is that history. A 5-minute period under 10 requests a minute is no evidence, for the baseline or the rule: a low-traffic service's p90 is one slow request.

• A series needs a full day of data (288 five-minute values) before it has a baseline; until then the rule is not applied to it and the rule table says baseline still building.
• The baseline is read once a day per series (one GetMetricData call carries a week of 50 series; R26 reads two per service, its faults and its request count), and never while the series is in alarm: an ongoing degradation does not become its own normal. A degradation that lasts long enough to fill 5 % of a week (about 8 hours) and then clears raises the baseline for the following week.
• A breach must last 3 consecutive 5-minute periods: a spike that recovers within 15 minutes raises nothing. One that lasts 15 minutes is an episode whether or not it recovers afterwards, and its incident stays on record.

An alert on an inventory resource (an instance, a load balancer, a database …) carries the resource's own service as its service: the first of its service, app, application or Name tags (keys in any case), else the resource's name. An instance tagged service=web therefore joins web's other alerts on one incident; the alert's summary still names the resource.

Application Signals and Container Insights on EKS

Application Signals services and Container Insights pods are not inventory resources. Their series belong to the EKS cluster they run in: an Application Signals environment eks:<cluster>/<namespace> names it, Container Insights' ClusterName dimension too. The cluster must be in the estate inventory (a tagged cluster is; Resource Explorer lists untagged ones) and inside the scope: its tags decide for every service and pod on it. Services of other environments (EC2, ECS, Lambda) are counted as series with no inventory resource. The alert's service is the series' own Service (R26, R27) or PodName (R30, the workload's name) dimension, and the cluster's name for R29, so a fault, a latency rise and a restarting pod of one service land on one incident. One resource carries at most 100 series a poll, shared round-robin across the rules, the rule with the fewest series first, so a cluster's many services never crowd out its one failed-node series (R29); the series past that are counted as over the 100 series per resource cap.

These rules follow the cluster's tags: on a cluster in scope, every Application Signals service and every pod is watched, whichever team's namespace it runs in. To keep to your own namespaces on a shared cluster, turn R26, R27 and R30 off and use the Kubernetes rules below, which read the namespace's labels.

R30 counts restarts in each 5-minute period (the CloudWatch agent reports the restarts since its previous collection): kubelet's crash-loop back-off caps at 5 minutes, so a container that keeps crashing restarts at least once in every period, and 3 periods in a row is 15 minutes of it.

Kubernetes state (R31 to R34)

A cluster's own state says what its metrics may not: a pod crash-looping, killed out of memory, stuck Pending, a node gone NotReady. Every poll, built-in detection reads the cluster through the workspace's read-only kubectl connection (kubectl get pods, get nodes, and get namespaces when a scope needs their labels — read verbs only, the same reader as the estate inventory, its namespace-by-namespace fallback included). This works for any cluster the connection reaches, EKS or not, and needs no Container Insights.

• One alert per workload, not per pod (pods change names): the workload is the pod's controller — a Deployment (through its ReplicaSet), StatefulSet, DaemonSet or Job; a bare pod is its own. Its service label comes from the workload's labels (app.kubernetes.io/name, then app, then k8s-app), else the workload's name. The alert also carries namespace, the workload (deployment=payments), one pod, the container, the cluster and resource_id k8s:<cluster>/<namespace>/<Kind>/<name>, the estate inventory's id for it. <cluster> is the connection's name (or a cluster value in its configuration), so two connections never share an id; the next inventory harvest removes the nodes an earlier version wrote as k8s:cluster/…, and an alarm under such an id keeps clearing the usual way. One alert per node for R34.
• R31 also counts a container between back-offs: after each crash the kubelet shows it terminated for a minute or so, then running again, and at the 5-minute back-off a poll can land there every time. A container that restarted 3 times or more and exited with an error since the previous read is a crash loop too; one lost with its node (reason Unknown after a reboot) is not.
• R32 counts an out-of-memory kill once, in the read after it happened. R33 waits 10 minutes (cluster autoscaling and a large image pull usually finish sooner) and says why the pod waits (Unschedulable: 0/3 nodes are available…, ImagePullBackOff). R34 waits 5 minutes, the time after which Kubernetes itself starts evicting a not-ready node's pods.
• Scope: a Kubernetes label counts as a tag. With a workspace scope, a pod is watched when it or its namespace carries every scope tag as a label (for example aiops-managed: "true" on the namespace); a node when it runs a watched pod, and while its own alert is open (Kubernetes evicts a not-ready node's pods after 5 minutes, the same moment R34 fires). Without a scope, everything the connection can read is watched.
• A failed read keeps every state as it is; the card's Other reads line says why. A read fails when the pods cannot be listed (forbidden, or no answer: each kubectl call gives up after 15 seconds, a cluster after 90, and a cluster whose pods do not answer is not asked for anything else), or when the call cap cut it short (40 calls, plus one per namespace listed on the connection). Clusters are read at the same time, not one after another. When only the namespaces cannot be listed, the scope is read from the pods' own labels and a workload already in alarm stays watched. A workload with no pod left, in a namespace the read listed and still inside the scope, has cleared: it resolves at once like a recovered one, so its incident can close on its own after the quiet period. A node that no longer exists, or a workload or node that left the scope, resolves at once with a note (nothing was seen to clear); a recovered one after three clean reads.
• At most 25 alerts open per connection: a broken cluster opens 25, and the card's Other reads line says how many more are not alerted; the rest open as those clear. The same cap holds for each account and region's scheduled maintenance and each account's provider issues.
• Every enabled read kubectl connection is read while built-in detection is on. To leave one out, set builtin_detection to false in its configuration (API: PATCH /api/aiops/{tenant_id}/datasources/{ds_id} with the whole current config and that key — a config patch replaces the config); the connection dialog has no box for it yet.

Scheduled maintenance (R35, R36)

AWS announces maintenance on a resource days or weeks ahead: an instance on degraded hardware is retired, a host is rebooted, a database gets an operating-system update. Nothing has failed yet, so these open a P4 incident on the resource itself (severity info: held, no paid analysis), in time to plan the change instead of meeting it. A mission-critical resource or a blast radius of 2 or more services raises the priority, as for any incident. P4 is our proposal: whether scheduled maintenance should be an incident at all, or a notice only, is still an open question. Once an hour, for each account and region with an in-scope EC2 instance or RDS database in the inventory, the evaluating AWS connection reads:

• ec2:DescribeInstanceStatus for the instances with a scheduled event (any instance state); one alert per instance and event code, with the window AWS gives (not before, not after) and the deadline. A description AWS marks [Completed] or [Canceled] is history and clears it.
• rds:DescribePendingMaintenanceActions; an action with a date — applied in the next maintenance window, automatically after a date, or forced on a date — is one alert per database and action. An action AWS files on an Aurora cluster is an alert on each of its instances in the inventory. An action with no date is offered, not scheduled, and raises nothing.

These are the EC2 and RDS APIs rather than the AWS Health API: they carry the same scheduled events for every account (the Health API needs a Business, Enterprise On-Ramp or Enterprise support plan) and name the resource directly. Only resources in the estate inventory and inside the scope are matched. One clean read ends an episode (the event completed or was canceled). A denied read keeps its rule's states and the card's Other reads line names the permission.

Provider health (R37)

When AWS itself has a problem, your alarms fire on the symptoms; R37 names the cause. Every poll, for each AWS account with an evaluating connection, built-in detection reads AWS's open issues for the services and regions the account's in-scope estate uses: the AWS Health API (health:DescribeEvents, on the global endpoint) through the connection, or, on an account without a Business, Enterprise On-Ramp or Enterprise support plan or a role without that permission, the public AWS Health Dashboard, which is not account-specific (the card's Other reads line then says needs health:DescribeEvents, and the Health API is asked again an hour later). An issue becomes an alert only where the estate is exposed: an in-scope resource of that service in that region (any service for an issue AWS files under multiple services).

• One P2 incident per issue (severity error): it is analysed like a critical alert, files its ticket and notifies. The alert's service is aws-<service>-<region> (aws-ec2-ap-southeast-1), its message names the exposed resources (up to 5, and how many), and it carries aws_service, health_event and which source answered. The automated triage's cloud-health step reads the same issue for any other incident it touches.
• The read is cached for 5 minutes per account (the public dashboard per set of regions and services), so a 1-minute poll does not read it more often. An issue AWS closes clears after three clean polls. An unreachable read keeps the states and the card says why.
• The read goes out from the platform itself, never from a private-network agent. On a network without internet, the Health API (health.us-east-1.amazonaws.com) needs an interface endpoint for AWS Health (com.amazonaws.us-east-1.health, offered cross-Region to Asia Pacific (Singapore); confirm it in your account with aws ec2 describe-vpc-endpoint-services --service-regions us-east-1 --service-names com.amazonaws.us-east-1.health) plus one for STS; the public dashboard has no VPC endpoint and needs internet egress. A read that gets no answer says which: no answer from the AWS Health API or no answer from the public AWS Health Dashboard, then what it needs. The automated triage's cloud-health step says the same.
• Which services count: those AWS Health reports on and the inventory records with an ARN — EC2, EBS, RDS, Elastic Load Balancing, Lambda, SQS, SNS, DynamoDB, S3, ECS, EKS, ElastiCache, API Gateway and Kinesis.

New rules start in shadow

A workspace that was already armed when a release added rules (R26 to R37 in 0.1.153) does not start them firing unseen. Each new rule watches in shadow from the first poll it covers something, for the workspace's shadow period (24 hours by default): what it would have fired is recorded under Would have fired (shadow), the rule table says New rule: shadow until and when, and the card names the rules. When its period ends the rule arms itself; what it already had in alarm then opens with no analysis ("in alarm before built-in detection armed"), as at a workspace's own arming. Arm now arms them at the next poll with the usual severity rules. A workspace that turns detection on after the upgrade arms every rule together.

What it reads, and what it never does

• The estate inventory. Resources come from the inventory harvest (Estate x-ray): every tagged resource, and since this release every EFS file system, OpenSearch domain, DynamoDB table, SQS queue, NAT gateway and API Gateway API, tagged or not, with the attributes the rules above need. A resource the inventory does not hold is not watched; the card counts the CloudWatch series it found with no inventory resource behind them. The list is re-read only when a harvest lands (daily by default, or Harvest now on the connection).
• ListMetrics with RecentlyActive=PT3H, one call per metric and namespace, cached for an hour: a resource is alive when CloudWatch lists it, whatever the inventory's flags say.
• GetMetricData, in batches of up to 500 metrics, every poll (5 minutes by default; the card offers 1, 5 or 15 minutes and the API accepts up to an hour — every 1-minute datapoint is evaluated whatever the poll).
• Never a write: no alarm, no anomaly detector, no metric stream, no ANOMALY_DETECTION_BAND.

Permissions

The CloudShell read role lists Lambda functions (lambda:ListFunctions, since 0.1.146). A page carries each function's environment variables: values under a secret-looking name (DB_PASSWORD, API_TOKEN) are masked before anyone or any model sees the result, the others are kept.

A missing permission is never a silent zero. The connection's line on Datasources and the card say needs cloudwatch:GetMetricData (or ListMetrics) for the account and region when AWS answers access denied. Refused credentials — deactivated or expired keys, an expired session, a clock out of step — read as the connection's credentials were refused (code) instead, and a role that cannot be assumed says so: there is no permission to add. Each is retried once an hour, and at once after a passing Test or a configuration change. When every always-published series of an account (running instances' CPU, for example) comes back empty, the card says evaluation returned no data instead of reporting the account healthy, and a listing cut at its page cap says listing truncated.

Which connection evaluates which account

Each (account, region) is evaluated once, through one connection: the account's inventory connection when its regions cover that region, otherwise the first connection by name that runs Built-in detection and covers it. A CloudShell read role is pinned to one region, so several connections into one account usually split the regions between them. The job is one of the Jobs this connection runs on the connection (config key builtin_detection; only an explicit false opts a connection out). Several workspaces on one shared account evaluate, and pay, separately; a workspace with workspace scope tags evaluates only the resources carrying all of them (an untagged resource is outside the scope), a connection with Only findings on resources tagged only the resources it harvested that carry that tag (exact value), and either never shows the "no inventory resource" count, which would reveal other applications' resources.

Cost

GetMetricData costs US$ 0.01 per 1,000 metrics requested, billed to the AWS account the connection reads. The card counts the metrics each poll requests, the totals for today and yesterday, and the estimated month per account at the current poll and coverage. Example: 300 resources × 3 metrics, every 5 minutes, is about 7.8 million metrics a month, about US$ 78 — the same order as 900 standard alarms. ListMetrics calls (about one per metric per region per hour) fall inside CloudWatch's free API requests for most accounts.

Max resources per poll (300 by default) caps the read; the cap is filled round-robin across kinds, so a fleet of instances never starves the databases and load balancers, and a resource with a rule in alarm is always read. What the cap leaves out is counted as not covered.

Noise controls

• M-of-N on 1-minute datapoints (EC2 instance metrics: 5-minute), as in the table.
• An evaluation delay of 600 s: the newest datapoint read is 10 minutes old, so data CloudWatch has not delivered yet is not read as missing (Datadog's CloudWatch monitors use 900 s).
• Missing data never breaches. A rate needs at least 10 requests (or invocations) a minute to count. A rate already in alarm keeps its state while traffic runs under that floor (no evidence either way), so a quiet night never clears a load balancer still failing every request; traffic that stops altogether is missing data and clears it.
• Clear after three clean polls in a row. A failed read is not a recovery: the state stays.
• A resolve from elsewhere never silences a rule still in alarm. Alerts resolve by fingerprint, whoever sends the resolve; a built-in alert closed that way while its rule is still breaching is sent again at the next poll and rejoins (or reopens) its incident.
• Warning rules are held. A warning opens or joins its incident with no investigation, no ticket, no push and no paid recheck (the recheck answers from the alert rows at no cost). Analyse on the incident, or a critical rule (or any other alert) joining it, starts the investigation and files the ticket as an open would. A workspace can let warning rules start an analysis (Warning rules start an analysis on the card).
• Grouping and correlation apply as to any alert. A held incident takes nobody in, held warnings buy no grouping turn, and the correlator sweep skips held built-in incidents. What was already in alarm when the workspace armed itself is a backlog of unrelated resources sent together, not a burst: it is never folded by shared cause (one incident per resource).
• Flapping: with a reopen window set, a held warning that clears and fires again inside it reopens the same incident (held again) instead of a new one per episode.

Shadow mode, and arming

Turning detection on starts a shadow period (24 hours by default): rules are evaluated and every pair that would have fired is recorded on the card with its observed value and, per rule, a hit total; nothing is opened. The shadow clock runs only while something is covered — with no inventory yet, a missing permission or every connection opted out it pauses, and the card says why.

When the period ends the workspace arms itself. Pairs already in alarm at that moment open their incidents with no analysis ("in alarm before built-in detection armed"), because nobody was watching: a critical one still files its ticket and sends the open notification (press Analyse to start the analysis), a warning stays held like any warning. New episodes after that follow the usual severity rules. The owner can Arm now (the confirmation says how many incidents open and how many start an analysis; what is already in alarm then follows the usual rules), Keep shadow indefinitely, or switch single rules off in the rule table. Arm now takes effect at the next poll — the card says it is waiting until then — and is not offered while Keep shadow is on (untick it first); ticking Keep shadow cancels an Arm now still waiting.

Turning it on and off

• Workspace owner (members.manage): the card's Turn on / Turn off. Turning off resolves every built-in alert still active, annotated built-in detection switched off — no longer watched; the confirmation lists the incidents first (their titles to a role that may read incidents, else their number). Turning on again starts a new shadow period.
• Per connection: untick Built-in detection under Jobs this connection runs. When no other connection evaluates that account and region, its built-in incidents resolve at the next poll with the same kind of note; the dialog lists them before you save.
• Stopping to watch is not a clear (turning off, a rule switched off, an opted-out connection, a resource leaving the workspace scope or the connection's tag scope): the incidents resolve with a note saying nothing was seen to clear, no recovery time is recorded, no verification runs after the close, and a ticket the incident has gets the write-back but stays open for its owner.
• Platform staff: the ops console's Built-in detection (AWS) switch stops it for every workspace. It freezes rather than resolves: alerts already firing stay open until it is back on, and the card says so. Disabling the AWS connector type itself freezes it the same way.
• A pair in alarm that no poll could read for 24 hours (a lost permission, an unreachable account) is listed on the card; it is never resolved by a failed read.

What an alert looks like, and how it sits beside your own alarms

Each alert carries alertname = rule ID and name (R11 ALB 5xx rate), service = the resource's name in the inventory, resource and resource_id = its ARN, resource_type, rule, severity, cloud=aws, account, region, metric_namespace, and the CloudWatch dimensions as labels (LoadBalancer=app/web-alb/…), the way the CloudWatch alarm webhook labels an alarm. The incident title reads Built-in rule R11 · ALB 5xx rate > 5 % for 5 of 5 min · web-alb; the alert says what it observed against the threshold and links the rule's source. Target-group rules key on the target group itself, so one load-balancer outage can open an ALB incident and a target-group incident.

Beside a native CloudWatch alarm on the same resource: the alarm webhook names the resource by its dimension value, a built-in alert by its name in the inventory. They agree for an EC2 instance, an RDS instance and a Lambda function (the inventory names those by their ID or name), and for an SQS queue or DynamoDB table without a Name tag: the two alerts land on one incident. An EFS file system, or a queue or table with a Name tag, is named after that tag in the inventory (shared-data against fs-0abc…); load balancers, target groups, OpenSearch domains and API Gateway stages are named differently too (web-alb against app/web-alb/<id>). Those open separate incidents unless the alarm's description declares service=<name>. Both carry the dimension labels, so a label_map, criticality_map or assignment_map entry can target either. The source builtin is the platform's own: a webhook sender that names it is recorded as generic.

Limits

• Latency: the 10-minute delay, plus the rule's window, plus up to one poll (5 minutes) from the first bad minute to an alert: about 11–16 minutes for a 1-of-1 × 1 min rule, 15–20 for 5 of 5 × 1 min (or 1 of 1 × 5 min), 20–25 for 10 of 10 × 1 min or 2 of 2 × 5 min, and 25–30 for 15 of 15 × 1 min or 3 of 3 × 5 min. About 25 minutes to clear. A native alarm is about 5 minutes.
• Low traffic: a rate needs 10 a minute; a quiet load balancer or function failing every request never fires.
• An NLB target group with no registered targets is not seen (its health metrics are not published). An ALB target group that lost every target is seen through R11, because the load balancer's own 503s count.
• Inventory: SQS retention is read for 20 queues per region per harvest, a different 20 each day, and what was read is kept (R18 reaches every queue after a few daily harvests); a REST API's rules need its name, from the list read. On a large account the per-harvest call cap reaches these list reads last, after every region's existing estate reads; the harvest notes say what it skipped. The first harvest after this release shows every RDS instance and every resource the new list reads describe as changed or new.
• Baselines exist for latency and fault rate (R26 to R28) only. Not in this phase: error bursts and traffic drops against a baseline, the further rules X1–X8, Application Signals dependency (caller-side) metrics, services outside EKS, and logs. A service that Application Signals sees only from its callers (no server-side instrumentation) has no service-level series for R26 and R27 to read.

Next steps

• Next: Synthetic checks
• All Connect your environment pages
• Documentation home