Set up the Test Lab on your own cluster

Provision an isolated test environment using your own Kubernetes cluster.

How-to · Updated 2026-10-17 · Describes release 0.1.164

What you are about to build

You give the platform one kubeconfig. It installs six demo estates, a full observability stack, and wires that stack's alertmanager to post alerts back into your Test Lab workspace. Nothing here asks you to write a manifest.

When it finishes, the lab looks like this:

The estates are real applications with real dependency chains, not stubs — that is the point. A case that kills a cache has to produce the same confusing symptoms your production incidents do, or the accuracy number it earns means nothing.

Before you start

Creating the lab workspace needs members.manage. Connecting, validating, provisioning, changing guards and teardown need datasources.manage. Launching or cancelling live runs and replays needs benchmark.manage; reading results and certifying needs benchmark.view.

A cluster you are willing to break. Faults are injected on purpose: pods are killed, deployments are scaled to zero, images are corrupted, network paths are cut. Use a cluster that hosts nothing you care about.

• Kubernetes 1.27 or newer.
• Free capacity: about 6 vCPU and 8 GiB, on top of anything else running. That is the six estates plus observability at rest. Add headroom if you plan to run cases in parallel — each extra shard is another copy of hotel.
• A default StorageClass. Each copy of hotel (the base and every shard) claims one 1 GiB volume for mongodb-rate, so the volume-full cases have a real claim to fill. The claims live in the estate namespaces and go with them on teardown. Every other estate is ephemeral.
• Outbound image pulls to us-central1-docker.pkg.dev, docker.io and gcr.io. All images are public; none are ours.
• No ingress controller, no load balancer, no public DNS. The lab talks to itself over cluster DNS, and the platform reaches it through your kubeconfig.

Step 1 — a kubeconfig the platform can build with

Provisioning creates namespaces, deployments, services, cluster roles and custom resources, so this credential has to be cluster-admin. The platform checks: it runs kubectl auth can-i '*' '*' and refuses to start if the answer is anything but yes. A narrower role fails before the first manifest is applied, rather than half way through.

kubectl create namespace lab-access
kubectl create serviceaccount lab-provisioner -n lab-access
kubectl create clusterrolebinding lab-provisioner \
  --clusterrole=cluster-admin \
  --serviceaccount=lab-access:lab-provisioner

# a non-expiring token: a projected one would expire mid-run
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: lab-provisioner
  namespace: lab-access
  annotations:
    kubernetes.io/service-account.name: lab-provisioner
type: kubernetes.io/service-account-token
EOF

Then build a kubeconfig from it:

SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl -n lab-access get secret lab-provisioner -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n lab-access get secret lab-provisioner -o jsonpath='{.data.token}' | base64 -d)

cat > lab-kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
clusters: [{name: lab, cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}}]
users: [{name: lab, user: {token: ${TOKEN}}}]
contexts: [{name: lab, context: {cluster: lab, user: lab}}]
current-context: lab
EOF

If the platform runs inside the same cluster, use https://kubernetes.default.svc as the server instead of the public API endpoint — the call never leaves the cluster.

Step 2 — register it as a write-role datasource

Open Agent quality → Test Lab, and on the Lab cluster card choose Connect lab cluster. The form opens on Kubernetes (kubectl + kubeconfig) with the role fixed to Lab provisioning and fault tests · write credential: paste the file, then Save lab connection. No automatic read probe runs; the saved credential's access is not yet verified. For this Test Lab setup, continue to Provision below: the provisioning operation and its log show whether the required write access works.

The role matters more than it looks. A write datasource is a remediation-class credential: it is excluded from every agent registry, so no investigation, x-ray scan or chat turn can see it or use it. Only the Test Lab's own provisioning path executes with it. The Lab refuses a read datasource for exactly this reason — a cluster-admin token an agent could reach for is a different product.

Step 3 — provision

Test Lab → Provision, choose the datasource, and confirm. A few minutes. This is the only click; everything in the next section happens inside it.

Before applying anything the platform lists every namespace on the cluster and flags the ones a lab does not normally have. That list is advisory — it exists so you notice before the estates land if you pointed at the wrong cluster.

Three settings decide whether alerts actually come back. Get these right and everything else is automatic:

• Reachable platform URL. The lab's alertmanager has to POST to this platform. If the lab sits behind a firewall, or this platform only accepts traffic from a CDN, the default public URL will not work — supply the address the lab can reach, which for a same-cluster lab is the internal service name.
• The ingest token. If your workspace already has a webhook token, hand it over here. Provisioning will not rotate a token in use: rotating it would 401 every real alert source you already have.
• Parallel shards. Leave at 1 unless you are running the catalogue for time. Each shard is a full copy of hotel. A re-provision keeps the count you set last, and deletes any shard above it that carries the lab's estate label; one it cannot prove is the lab's stays, and the log names it.

What Provision does for you

Nothing below is a step you take. It is what the one click does, listed so you know what lands on your cluster:

• The six estates and the observability stack, as above.
• Alertmanager rewritten to post at this platform, then restarted so it reads the new config.
• Chaos Mesh and LitmusChaos, installed from the versions this platform ships, each only when the cluster does not run it yet. One that is already running is left alone: a re-provision must not restart the injector under a campaign in flight. LitmusChaos counts as running when its operator does — its CRDs alone are not enough.
• An event guard on the estates. An admission policy, matched to the estate namespaces only, refuses the chaos engines' own events there, so the agent under test cannot read which fault was injected; every other event passes. It needs Kubernetes 1.30 or later. Without it the provision still succeeds, and runs remove those events while each case is in flight — the log says which applies.
• The lab Git server, for the lab's GitOps cases; its data is ephemeral.
• The lab cluster connection, scoped to the estates. Typed restarts the change agent proposes may target any estate namespace, every hotel shard included. A Test on that connection confirms only the namespace its kubeconfig context names, if any, until the next provision.
• Your lab datasources, registered. The read-only Kubernetes identity, plus Prometheus, Loki, Tempo and Alertmanager. Each URL is probed first and skipped if the platform cannot reach it, so you never end up with a datasource that looks connected and is not.

Two things to know about that last one. These are cluster-internal addresses, so they only work when this platform runs in the same cluster as the lab; if it does not, the provision log names each one it skipped and you add reachable addresses yourself. And the hostnames it did register are added to your workspace's allowed-hosts list — without that, the datasource works for the agent but its Test button reports the SSRF guard, because the guard's job is to stop anyone pointing a datasource at an internal address that was never meant to be reached.

If a chaos install fails, the provision still succeeds. The estate is up and the cases that scale, patch and delete still run; only fault injection is missing, and the log says so. Re-run provision to retry it.

Verify

• Test Lab → Provision log ends with provision complete — estate is feeding alerts at ….
• All four namespaces report every pod running.
• Within a few minutes, Incidents is quiet. If a "storefront is not answering" incident opens and stays open, an estate did not come up — the probe is telling you which one.
• Settings · Data in → Data sources lists the lab's read connections; review their read-test results. The separate write credential is checked by the provisioning operation, not by a green read test.

Then run two or three cases before the full catalogue. A run that scores zero usually means the alerts never arrived, not that the agent was wrong.

Tearing it down

Test Lab → Teardown removes what provisioning created, and only that:

• the estate namespaces — every hotel shard the lab made included — observability and the lab's Git server (a shard-named namespace without the lab's estate label stays, and the log names it)
• the read-only roles it created outside any namespace, for the agent and the metrics scraper, and the estates' event guard
• LitmusChaos, when provisioning installed it (its CRDs stay; the log shows the command that removes them)
• the datasources it registered for you, and their entries in your allowed-hosts list

It does not touch your lab kubeconfig, past runs, or their graded results. A namespace still terminating three minutes after its delete ends the job done with a WARNING naming it: the delete was accepted, so run Teardown again to confirm it is gone. When a step fails — a delete the cluster refused, a namespace still terminating on that second run or whose deletion the cluster reports blocked, a route of yours still using a lab connection — the job ends failed and names what is left. Fix the cause and run Teardown again: it picks up where the last one stopped.

Chaos Mesh is deliberately left installed. Provisioning only installs it when the cluster has none, so it may well have been there before this lab — and it is cluster-scoped: uninstalling deletes the chaos CRDs, and with them any experiment anyone else on that cluster is running. The teardown log shows the command instead, with a copy button, so removing it is a decision you make:

helm uninstall chaos-mesh -n chaos-mesh &&   kubectl delete ns chaos-mesh &&   kubectl get crd -o name | grep chaos-mesh.org | xargs -r kubectl delete

The CRD line matters — helm uninstall leaves custom resource definitions behind by design. If Argo CD installed Chaos Mesh there is no helm release, and self-heal puts back whatever you delete: remove its Application from Git instead, after adding the resources-finalizer.argocd.argoproj.io finalizer to it, or the prune leaves the workloads and CRDs running.

Re-provisioning is idempotent, so tearing down between campaigns to stop paying for the nodes is a reasonable habit.

Next steps

• Next: Benchmarks and scores
• All Work with the agents pages
• Documentation home