Skip to content
AI Developer Toolkit

Find your level — free Buy the Toolkit Buy Toolkit Buy the Toolkit Buy Toolkit Open the guides Read
Navigation menu
Buy the Toolkit Find your level — free Buy the Toolkit Open the guides Sign in

Kubernetes Orchestration Patterns

PL

Level 2 How-to Updated

Kubernetes orchestration patterns for AI agents are prompts and gates that make Claude Code, Codex, or Cursor write Deployments and Helm charts that encode real operational limits: probe timing, memory headroom, disruption budgets, and DNS egress. But a manifest that applies cleanly can still cause an outage, so every change passes schema, server dry-run, and diff checks before merge.

Your rollout just took the API down. The readiness probe went green before the Postgres pool finished warming, so traffic moved onto pods that returned 500s, and the manifest came from an agent that never knew the pool needs eight seconds. This page is for the developer or tech lead who owns that manifest and wants the next one proven by a script, not by a person reading 400 lines of YAML.

What you’ll walk away with from these Kubernetes patterns

Section titled “What you’ll walk away with from these Kubernetes patterns”
  • A cluster rules block for CLAUDE.md, AGENTS.md, or a Cursor rule, so the agent starts from your constraints instead of its defaults.
  • scripts/verify-k8s.sh, a gate that renders the chart and checks schema, invariants, server-side admission, and the live diff.
  • Copy-paste prompts for a probe-safe Deployment, a Helm chart without Bitnami dependencies, a CrashLoopBackOff diagnosis, and a resource-limit tune from real usage.
  • A read-only Kubernetes MCP setup for all three tools, so diagnosis runs against live cluster state.
  • Recovery steps for six failures agent-written manifests commonly cause.

Give the agent your cluster rules first

Section titled “Give the agent your cluster rules first”

Agents fill every gap in a prompt with a training-data default: initialDelaySeconds instead of a startupProbe, a latest image tag, a default-deny NetworkPolicy with no DNS rule. Write the rules down once, in the file each tool reads on every session: CLAUDE.md for Claude Code, AGENTS.md for Codex, and a rule under .cursor/rules/ for Cursor.

## Kubernetes rules (k8s/ and charts/)
- Every container: requests and limits for CPU and memory; memory limit >= 1.25x observed P99.
- Slow-starting apps get a startupProbe; never gate readiness with initialDelaySeconds.
- Probes use named ports. Liveness never checks downstream dependencies.
- Pods run as non-root, readOnlyRootFilesystem: true, drop ALL capabilities, seccomp RuntimeDefault.
- Images are pinned by version tag or digest. Never `latest`.
- A PodDisruptionBudget sets exactly one of minAvailable / maxUnavailable.
- A default-deny NetworkPolicy always allows egress to kube-dns on UDP and TCP 53.
- No Bitnami dependencies (charts.bitnami.com or oci://registry-1.docker.io/bitnamicharts). Postgres comes from the CloudNativePG operator.
- If an HPA manages a Deployment, do not set spec.replicas in the Deployment.
- Done means: scripts/verify-k8s.sh exits 0. Paste its summary into the PR.

The last rule is the one that changes the review. It turns “looks fine to me” into an exit code the agent must produce before it hands the work back.

Prove a manifest without reading every line

Section titled “Prove a manifest without reading every line”

Reading generated YAML line by line does not scale, and it misses the failures that matter, because a missing DNS rule looks like nothing at all. Instead, render what will actually reach the cluster and let four layers check it. Run the script in your terminal and in CI.

#!/usr/bin/env bash
# scripts/verify-k8s.sh  (terminal or CI; needs helm, kubeconform, kubectl)
set -euo pipefail
CHART=charts/orders-api
VALUES=$CHART/values-production.yaml
OUT=.rendered; rm -rf "$OUT"; mkdir -p "$OUT"


# 1. Render exactly what will be applied
helm lint "$CHART" -f "$VALUES"
helm template orders-api "$CHART" -n production -f "$VALUES" > "$OUT/all.yaml"


# 2. Schema: built-in kinds plus CRDs (such as the CNPG Cluster)
kubeconform -strict -summary \
  -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  "$OUT/all.yaml"


# 3. Invariants a schema cannot express
fail() { echo "FAIL: $1"; exit 1; }
grep -Eq 'image: .*:latest' "$OUT/all.yaml" && fail "image pinned to latest"
# Assumes no registry port: registry.internal:5000/orders-api (no tag) passes.
grep -Eq '^\s*image: "?[^:@"]+"?\s*$' "$OUT/all.yaml" && fail "image with no tag or digest"
# orders-api warms a Postgres pool, so its Deployment must have a startupProbe.
# Drop this check for charts whose apps start fast.
if grep -q 'kind: Deployment' "$OUT/all.yaml"; then
  grep -q 'startupProbe:' "$OUT/all.yaml" || fail "Deployment without startupProbe"
fi
if grep -q 'kind: NetworkPolicy' "$OUT/all.yaml"; then
  grep -q 'k8s-app: kube-dns' "$OUT/all.yaml" || fail "NetworkPolicy without DNS egress"
fi
grep -Eq 'charts\.bitnami\.com|bitnamicharts' "$CHART/Chart.yaml" && fail "Bitnami dependency"
grep -Eq 'image: .*bitnami' "$OUT/all.yaml" && fail "Bitnami image"


# 4. Admission and drift against a real API server (staging or kind)
kubectl apply --dry-run=server -f "$OUT/all.yaml"
set +e; kubectl diff -f "$OUT/all.yaml" > "$OUT/cluster.diff"; rc=$?; set -e
[ "$rc" -le 1 ] || fail "kubectl diff errored (rc=$rc)"
echo "OK: $(grep -c '^kind:' "$OUT/all.yaml") objects; diff in $OUT/cluster.diff"

Each layer catches a different class of defect:

LayerCatchesMisses
helm lint and helm templateTemplate errors, the chart’s own fail checksAnything valid but wrong
kubeconform -strictWrong apiVersion, unknown or misspelled fieldsBehaviour: probe timing, DNS
Invariant grep linesYour team’s rules, such as a pinned image tag and DNS egress presentRules you have not written down
kubectl apply --dry-run=serverAdmission: Pod Security Standards, policy webhooks, PDB validation, missing CRDsRuntime behaviour
kubectl diffWhat actually changes in the clusterNothing; a person reads this one

kubectl diff exits 1 when differences exist and above 1 on error, which is why the script tolerates exactly 1. cluster.diff is the one artifact a person reads: it is short, and it shows the change in cluster state rather than the template that produced it. Attach it to the PR with the script’s summary line, as the evidence bundle describes. The owner of the service signs off on the diff, not on the YAML.

After the merge, the gate moves to the rollout. kubectl rollout status deployment/orders-api -n production --timeout=5m fails the pipeline if new pods never become Ready. For canary analysis and automated rollback, use progressive delivery.

Run the agent on a manifest directory

Section titled “Run the agent on a manifest directory”

The prompts below work in all three tools. The difference is how you run them against a directory of manifests, and how you keep the agent away from the cluster while it writes.

Start claude in the repository root and paste a prompt. Claude Code reads CLAUDE.md and your existing kustomization.yaml or chart, so new objects follow the same labels and overlays.

For a read-only audit in CI or a script, use print mode with plan permissions, so the agent can read files but not edit them:

Terminal window
claude -p --permission-mode plan --output-format json \
  "Audit charts/orders-api against the Kubernetes rules in CLAUDE.md. List violations by severity with file and line."

The bare claude "query" form opens an interactive session seeded with the prompt. It is not a batch command. From v2.1.283 (both release channels on 2026-10-12), interactive sessions start in auto mode, where a classifier approves most actions. To keep the agent from changing a live cluster, add deny rules to .claude/settings.json, such as "deny": ["Bash(kubectl apply *)", "Bash(kubectl delete *)"], and start the session with a kubeconfig context bound to a read-only RBAC role. A prefix rule catches only the plain spelling of a command, so the RBAC role is the control that holds.

Generate a Deployment that does not race its own probes

Section titled “Generate a Deployment that does not race its own probes”

A frequent defect in an agent-written Deployment is a readiness probe that passes before the app is ready. Put the warm-up window into the prompt, and the startupProbe gates the other probes correctly.

The part to look for in the result is the probe block:

# k8s/base/deployment.yaml (container excerpt)
startupProbe:            # gates liveness and readiness while the pool warms
  httpGet: { path: /health/startup, port: http }
  periodSeconds: 2
  failureThreshold: 30   # up to 60 s
readinessProbe:
  httpGet: { path: /health/ready, port: http }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /health/live, port: http }
  periodSeconds: 10
  failureThreshold: 3

If the agent returns initialDelaySeconds on the readiness probe instead, reject it. initialDelaySeconds is a fixed guess that is either too short on a cold node or wasted time on a warm one. A startupProbe waits for the real signal and stops liveness from killing the pod during warm-up.

Lock down networking without breaking DNS

Section titled “Lock down networking without breaking DNS”

Ask for the Service, Ingress, and NetworkPolicy in the same prompt, so the workload is exposed and restricted at once. A rule agents often omit is DNS:

# k8s/base/networkpolicy.yaml (egress excerpt)
egress:
- to:
  - namespaceSelector: {}
    podSelector:
      matchLabels:
        k8s-app: kube-dns
  ports:
  - { protocol: UDP, port: 53 }
  - { protocol: TCP, port: 53 }

Without it, pods start, cannot resolve any hostname, and log no such host errors that look like an application bug. The invariant line in verify-k8s.sh fails the build instead. Check the k8s-app: kube-dns label against your cluster with kubectl get pods -n kube-system --show-labels, because some distributions label CoreDNS differently.

Parameterize the manifests into a Helm chart

Section titled “Parameterize the manifests into a Helm chart”

When the same manifests ship to dev, staging, and production, have the agent extract a chart with values validation. Correct one training-data default first: agents still reach for Bitnami dependencies. Bitnami moved its versioned container images out of docker.io/bitnami into the unmaintained bitnamilegacy repository from 28 August 2025, and its packaged charts stopped receiving updates (Bitnami announcement). A pin copied from an old example can resolve to an image that no longer exists.

For Postgres, install the CloudNativePG operator once per cluster and let the app chart render only a Cluster resource. Do not bundle an operator as a subchart of an app, because its CRDs and cluster-wide RBAC then share the app’s release lifecycle.

Terminal window
# Platform team, once per cluster (terminal)
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --install cnpg --namespace cnpg-system --create-namespace cnpg/cloudnative-pg

The validation helper makes a bad values file fail at helm template, which is the first layer of the gate:

# charts/orders-api/templates/_helpers.tpl (excerpt)
{{- define "orders-api.validateValues" -}}
{{- if not .Values.image.repository }}
  {{- fail "image.repository is required" }}
{{- end }}
{{- if and .Values.ingress.enabled (not .Values.ingress.hosts) }}
  {{- fail "ingress.hosts must be set when ingress.enabled is true" }}
{{- end }}
{{- end }}

Scale and drain safely with an HPA and a PDB

Section titled “Scale and drain safely with an HPA and a PDB”

Generate the HorizontalPodAutoscaler and the PodDisruptionBudget together. The PDB has a sharp edge: it accepts minAvailable or maxUnavailable, never both, and the API server rejects a PDB that sets both. maxUnavailable: 1 is usually the better choice with at least two replicas, because it keeps working when the HPA changes the replica count. At one replica, any PDB either blocks drains or allows downtime.

# pdb.yaml — exactly one of minAvailable / maxUnavailable
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: orders-api, namespace: production }
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: orders-api }

With GitOps, the HPA and Argo CD fight over spec.replicas unless you tell Argo CD to ignore it. Add this to the Application:

spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers: [/spec/replicas]
  syncPolicy:
    syncOptions: [RespectIgnoreDifferences=true]

ignoreDifferences on its own only hides the diff: if spec.replicas is still in Git, every sync (and every self-heal) writes it back over the HPA. So remove spec.replicas from the Deployment in Git, as the rules file says, or keep the RespectIgnoreDifferences=true sync option shown above so a sync leaves the replica count alone.

For traffic splitting and canaries, progressive delivery covers the analysis step that decides whether to promote. When you generate Istio routing, check which API versions your mesh serves with kubectl api-resources --api-group=networking.istio.io before you accept the apiVersion the agent picked.

Diagnose a failing pod from live cluster state

Section titled “Diagnose a failing pod from live cluster state”

Without cluster access, an agent answers “why is this pod restarting?” with a generic checklist. With the community Kubernetes MCP server (npm kubernetes-mcp-server, version 0.0.67 on 2026-09-26), it lists pods, reads logs, and pulls events itself. Its runtime settings live in a TOML file; the CLI keeps only --config, --config-dir, and --version. Start it read-only:

~/.config/k8s-mcp.toml
read_only = true
toolsets = ["core", "config", "helm"]

Also point the server at a kubeconfig context bound to a read-only RBAC role. read_only = true limits the server’s tools; RBAC limits what any credential leak can do. Keep toolsets to what diagnosis needs, because every enabled tool’s definition takes up context in each session. claude mcp add writes to local scope by default (Claude Code 2.1.283); add --scope project to write .mcp.json, which you commit so the whole team gets the same server.

Terminal window
claude mcp add kubernetes -- npx -y kubernetes-mcp-server@0.0.67 --config "$HOME/.config/k8s-mcp.toml"

No cluster access from the agent? Run kubectl describe pod POD -n production and kubectl logs POD -n production --previous yourself and paste both outputs above the same prompt. More infrastructure servers, and when a skill fits better than a server, are on infrastructure MCP servers.

Tune resource limits from observed usage

Section titled “Tune resource limits from observed usage”

Agent-chosen limits are guesses. Replace them with numbers from your own metrics.

Where the metrics come from, and the alerts that tell you a limit is too tight, are on monitoring and logging patterns.

What breaks when an agent writes your Kubernetes config?

Section titled “What breaks when an agent writes your Kubernetes config?”

  1. CrashLoopBackOff right after a rollout. The liveness probe kills the pod before it finishes booting. Confirm with kubectl describe pod POD -n production: the events show Liveness probe failed followed by a restart. kubectl logs POD -n production --previous shows the boot sequence, cut short. Recover: move the boot guard into a startupProbe, and add a startupProbe check to verify-k8s.sh if it is not already there.

  2. OOMKilled under load. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled. Recover: raise the memory limit above observed P99 with the resource prompt above and real data. Do not double the number to make the alert stop.

  3. A node drain hangs. The PDB leaves zero disruption budget, for example minAvailable: 3 with three replicas. Recover: switch to maxUnavailable: 1. A PDB that sets both fields never reaches the cluster, because the server dry-run in the gate rejects it.

  4. ImagePullBackOff. The tag does not exist, the registry needs imagePullSecrets, or a chart still points at a removed Bitnami image. Recover: confirm the tag in your registry, attach the pull secret, and replace the Bitnami dependency as shown above.

  5. Argo CD stuck OutOfSync or Progressing. OutOfSync that returns after every sync is usually the HPA and Git fighting over spec.replicas: add the ignoreDifferences entry, and either remove spec.replicas from the Deployment in Git or set RespectIgnoreDifferences=true, so a sync does not reset the replica count. Progressing that never ends means the Deployment never becomes Ready: go back to items 1 and 2.

  6. Your emergency kubectl rollout undo reverts itself. With selfHeal: true, Argo CD restores whatever Git says within minutes. Recover: revert the commit in the config repository. That is the rollback; kubectl is not. Rehearse the path as described in rollback pipelines.

A green gate with a broken production is the one silent failure here. It happens when the staging cluster used for server dry-run lacks a CRD, a policy, or a Pod Security label that production has. Keep the two in step, and treat a new admission rejection in production as a gap in staging.

Where to go next with Kubernetes and agents

Section titled “Where to go next with Kubernetes and agents”
Docker and container patterns Build and prove the non-root, pinned images these manifests deploy.
Docker and Kubernetes containerization The end-to-end workflow from Dockerfile to hardened manifests, with the Docker MCP gateway.
CI/CD pipeline patterns Run verify-k8s.sh in CI and bump image tags in the GitOps repository.
Infrastructure as code patterns Apply the same render-validate-diff gate to Terraform and the clusters themselves.
Reviewing an agent's pull request Decide what a person must read in an agent's PR, and what the gates prove instead.
Choosing a model Start on the tool's default model and switch only when your evals say so.

Did this guide help you?

One tap, no sign-in.

not at all a lot
All 577 guides Free resources: Compare tools Glossary By role

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings llms.txt pricing.md RSS