Patching and cluster upkeep

The cloud team owns cluster version, capacity and certificates; the release team owns application images and dependencies.

How-to · Updated 2026-10-19 · Describes release 0.1.164

Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).

The cloud team owns cluster version, capacity and certificates; the release team owns application images and dependencies. EKS Auto Mode manages nodes and built-in components, but it does not remove the need for an operator to initiate Kubernetes version upgrades or verify application compatibility.

Monthly image and dependency review

ECR enhanced continuous scanning is enabled for every repository in the publishing registry, verified on 6 October 2026. Review the findings for the running image tag, not merely the newest built image. Customer cache-registry settings must be reviewed in their own account; upstream scanning is not proof of a customer-side configuration.

The following read is quoted from the recorded registry review. The registry operator sets AIOPS_IMAGE_REPOSITORY and AIOPS_IMAGE_TAG to the published repository and running tag, in the registry's account/region:

aws ecr describe-image-scan-findings \
  --repository-name "$AIOPS_IMAGE_REPOSITORY" \
  --image-id "imageTag=$AIOPS_IMAGE_TAG" \
  --query imageScanFindings.findingSeverityCounts

That query returns the scan's summary field, not a reliable complete vulnerability total: the security review found it represented only the first page. For decisions, read the complete paginated enhanced findings or the Inspector results. Keep severity, status, package, fixed version, scan time, image digest and counting method. Compare like measures month to month.

The 0.1.108 security receipt reports the Helm update removed 111 of 113 Helm findings from the earlier image, including its 13 critical findings; two high findings in its compiled cryptography dependency remained. The 0.1.109 and 0.1.110 images carry the same dependency set; this does not establish a new scan. These are dated scan results, not a guarantee that the current registry scan is clean. See the full dependency snapshot.

Release team: update only reviewed pins/constraints, build a new version, run the project release checks and scan the result. Publish image/chart and the finding receipt. Customer approver: use Upgrade and retain the previous image/chart for Rollback. An enhanced scanner is detection, not an automatic patch or acceptance gate.

Kubernetes minor-version upgrade

This procedure is code-derived and not drilled on a customer cluster. The fact-check snapshot found Kubernetes 1.36 and passing 1.37 upgrade insights on the hosted environment; that is not compatibility proof for a customer or for self-managed controllers.

1. Cloud team checks the installed version, AWS support horizon and upgrade insights for the intended next minor. Review deprecated APIs, add-ons, self-managed controllers, applications and disruption budgets.
2. Release/cloud team changes cluster_version in the customer branch's infrastructure/iac/envs/<slug>/env.hcl, one minor at a time.
3. Use the approved infrastructure plan/apply workflow for env=<slug>, unit=eks, or the customer deployment pipeline. The latter plans and applies every unit, so read all changes before releasing it.
4. Auto Mode handles its networking, load-balancer, DNS, kube-proxy, provisioning and EBS components and replaces nodes as they expire. Operators still maintain application charts, self-managed components and EKS add-ons.
5. Verify nodes, application health, volume attachment, metrics, runner claims, login and audit sealing; then run the tenant smoke and record the actual downtime. Treat that as a new measured receipt.

Use the installed account's maintenance arrangement; a hosted-environment hold or sync window is not a customer-cluster control. Do not describe Kubernetes upgrades as rehearsed until that evidence exists.

Node rotation and capacity

Customer-template clusters use built-in Auto Mode node pools. AWS manages their replacement on new node images (roughly weekly) and 14-day expiry; the built-in pools have a 10% disruption budget and no customer scheduling window. The platform chart does not supply a custom NodePool. A hosted custom-pool disruption hold does not apply to these customer pools, and expiry/unhealthy replacement can bypass a voluntary-disruption hold.

Review Ready nodes, draining/expiring nodes, pending work and requested CPU/memory against available capacity and quota. The single-replica control plane can move only to a node able to attach its /data volume in that zone. A PodDisruptionBudget can extend draining up to the Auto Mode node's 24-hour termination grace; understand these budgets before a node change. The control plane has a Recreate interruption when moved and no delivered multi-AZ failover.

Also check RDS connections, free storage and the 100 GB default autoscaling ceiling over 20 GB. Confirm SNS alarm recipients; existing alarms without a confirmed destination do not provide an on-call response. Use Monitoring and Incident response for capacity failures.

Certificates and expiring credentials

Customer installation uses ACM at the ALB, no cert-manager or external-dns. DNS-validated Amazon-issued ACM certificates renew while in use and their validation CNAMEs still resolve. Keep those records after installation, including when the hostname belongs to another DNS owner. Imported certificates need the owner's renewal procedure; do not assume ACM renews them.

Monthly inspect certificate status/expiry, DNS validation and renewal owner; route cloud renewal warnings to the on-call tool. Review PAT, agent-token, gateway credential and licence expiries in the same calendar. Where an optional database/OS maintenance action is pending, review it with the cloud team and book an approved window rather than assume the platform deploy handles it. Customer upkeep does not depend on the hosted environment's power-down schedule.

Sources: Security review §6 and AR-12, Availability and maintenance, Deployment runbook §1.5 and §3, customer EKS template, customer configuration, Dockerfile and Python constraints.

Next steps

• Next: Recurring checklist
• All Install and operate the platform pages
• Documentation home