Proactive monitoring
Incidents without alerts of your own — the platform watches endpoints, metrics and the cloud's own state, and raises outages and degradation by itself, shadow first.
Concept · Updated 2026-10-28 · Describes release 0.1.164
An alert from your own monitoring is one way into an incident, not the only one. Many estates have no alarm at all on most of their resources, so the platform also watches by itself: it reads the metrics, health signals and state your cloud already publishes and raises what it finds as an ordinary alert into the same pipeline. Everything here is deterministic — no model decides anything — and read-only: nothing is created in your account.
What counts as an incident
A service that is down or degraded is an incident, whatever noticed it first: your own alert, a ticket, or one of the platform's own detectors below. Degraded means slower or failing more than it normally does — latency or a fault rate above the service's own baseline — not only an outage. A failure that recovers still raises once it has lasted as long as its detector asks — a synthetic check confirmed by its retry, a CloudWatch rule's M of N periods (3 × 5 minutes for the baseline rules R26–R28, so a spike shorter than 15 minutes raises nothing): its incident shows when it started and that it cleared, and the analysis decides whether it matters.
The detectors
| Detector | What it watches | Source of the alert | Starts as |
|---|---|---|---|
| Synthetic checks | An HTTP or TCP probe of an endpoint every 1 to 10 minutes | synthetic |
your own checks: armed; checks suggested from the service graph: in trial until kept |
| Built-in detection, outages | CloudWatch rules R01–R25, R29, R30 over the estate inventory: status checks, unhealthy targets, 5xx rates, throttles, storage, failed EKS nodes, restarting pods | builtin |
shadow, then armed |
| Built-in detection, degradation | R26–R28: an Application Signals service's fault rate and latency, an ALB's target response time, each against its own 7-day baseline | builtin |
shadow, then armed |
| Built-in detection, Kubernetes state | R31–R34 through a read-only kubectl connection: crash-looping and out-of-memory containers, pods Pending over 10 minutes, nodes not Ready over 5 minutes | builtin |
shadow, then armed |
| Built-in detection, scheduled maintenance | R35, R36: an EC2 retirement, stop, reboot or maintenance event, an RDS maintenance action with a date — hourly, on the resource itself | builtin |
shadow, then armed; a P4 incident (our proposal) |
| Built-in detection, provider health | R37: an open AWS Health issue for a service and region where the estate has an in-scope resource of that service | builtin |
shadow, then armed; a P2 incident, analysed |
Each detector keeps to the workspace scope (its tags) and, for AWS, to the connection's Only findings on resources tagged scope: another team's resource on a shared account is neither read nor alerted on. In Kubernetes a label counts as a tag, on the pod or its namespace. One exception: the EKS rules R26, R27 and R30 follow the cluster's tags, so on a shared cluster in scope they watch every team's services and pods; turn them off there and keep to the Kubernetes rules, which read the namespace labels.
Alert is optional: none of these needs an alarm, a Lambda function or an agent of your own in
the account. They need a read connection — AWS for the CloudWatch rules, scheduled maintenance
and provider health, kubectl for the Kubernetes rules — and, for AWS, the estate inventory that
lists the resources to watch. Built-in detection is off until a workspace owner turns it on
(Settings · Data in → Datasources (in the product: /portal/datasources) → Built-in detection (AWS)); synthetic
checks have their own page.
Shadow first
A detector never starts opening incidents unseen. Built-in detection evaluates for a shadow period (24 hours by default) after a workspace owner turns it on, recording what it would have fired; then it arms itself, or the owner presses Arm now. A rule a release adds to a workspace that is already armed watches in shadow on its own for the same period (New rules start in shadow). A suggested synthetic check runs in trial until a person keeps it.
A built-in warning opens or joins its incident with no paid analysis until a person presses Analyse (or the workspace lets warning rules start one); a critical rule, and the provider health rule's error (P2), is analysed like any alert. Every number a detector uses, and why it has that value, is in Thresholds and why.
What it costs
The metric rules read CloudWatch (GetMetricData, US$ 0.01 per 1,000 metrics, billed to the
account read); the card shows the metrics per poll and an estimated month. A baseline is read once
a day per series. Synthetic checks cost one request per interval to your endpoint.
Platform metrics
aiops_detection_episodes_total counts the episodes each detector began, sent or recorded in
shadow, and aiops_detection_reads_total its reads by outcome (see the platform monitoring
reference).