Production Performance Optimization

Level 4 How-to Updated 2026-10-12

Production performance tuning with AI agents means connecting Claude Code, Codex, or Cursor to read-only Kubernetes, Postgres, and metrics MCP servers, so the agent correlates pod usage, traces, and slow queries into a cited diagnosis. The agent proposes the fix; a load test, a canary, and a before-and-after comparison against thresholds set in advance decide whether it ships.

It is 19:40 on a Friday. Checkout p99 has gone from about 200 ms to 3 seconds since the evening peak started, and the on-call channel is full of alerts. You have three dashboards open and cannot tell which signal is the cause and which are symptoms.

This guide is for backend developers and SREs who carry the pager, and for the tech lead who signs off a performance change before it reaches every replica. It assumes you already have metrics and traces; if you do not, start with monitoring and observability with AI.

For making Claude Code itself faster and cheaper, see Claude Code performance optimization.

What you’ll walk away with from agent-assisted performance tuning

A read-only MCP setup for Kubernetes, Postgres, and Grafana or Datadog, with the install command for each tool
Four copy-paste prompts: diagnose a p99 regression, size connection pools, fix a flapping autoscaler, and write the before-and-after verification report
An acceptance gate that proves a change worked without anyone reading the agent’s reasoning line by line
The failure modes of agent-assisted tuning, and how to recover from each

Which MCP servers does performance work need?

Performance diagnosis needs three signals: what the cluster is doing, what the database is doing, and what the users see. Each has a maintained MCP server with a read-only switch. Use the switch: the server acts with the credentials you give it, and no agent-side permission setting narrows that.

Signal	Server	Package	Read-only switch
Pods, HPAs, events, `pods_top` usage	Kubernetes MCP Server (`containers` org)	npm `kubernetes-mcp-server` 0.0.67	`read_only = true` in its TOML config
Slow queries, `EXPLAIN` plans, connection health	Postgres MCP Pro (Crystal DBA)	PyPI `postgres-mcp` 0.3.0	`--access-mode=restricted`
Prometheus and Loki queries	Grafana MCP (Grafana Labs)	PyPI `mcp-grafana` 1.6.0	`--disable-write`
Metrics, traces, monitors	Datadog MCP (managed, remote)	none: `https://mcp.datadoghq.com/v1/mcp`	scope the Datadog account you sign in with

Versions were read from npm and PyPI on 2026-09-26. mcp-grafana ships new releases every few days, so check PyPI for the current version before you pin one (a 2.x major line began on 2026-10-01; confirm the --disable-write flag in its README). For alternatives and more backends, see database MCP servers and observability MCP servers.

Configure the Kubernetes server as read-only

The Kubernetes server reads its runtime settings from a TOML file; its CLI keeps only --config, --config-dir, and --version. Save this as ~/.config/k8s-mcp.toml and point kubeconfig at a kubeconfig whose service account can only get, list, and watch:

read_only = true
toolsets = ["core"]
kubeconfig = "/home/you/.kube/perf-readonly.config"

# Keep Secrets out of the agent's context even if RBAC allows reading them
[[denied_resources]]
group = ""
version = "v1"
kind = "Secret"

The pods_top tool reads from the Kubernetes Metrics Server, so it returns nothing on a cluster that does not run it.

Add the servers to your agent

The servers are the same for all three tools; only the registration differs.

Run in your terminal. -e sets an environment variable for a stdio server, and everything after -- is the server’s own command:

claude mcp add kubernetes -- npx -y kubernetes-mcp-server@latest --config ~/.config/k8s-mcp.toml
claude mcp add postgres -e DATABASE_URI="$PERF_DB_URI" \
  -- uvx postgres-mcp --access-mode=restricted
claude mcp add grafana -e GRAFANA_URL=https://grafana.example.com \
  -e GRAFANA_SERVICE_ACCOUNT_TOKEN="$GRAFANA_RO_TOKEN" -- uvx mcp-grafana --disable-write
# Datadog instead of Grafana (US1 site; other sites change the host):
claude mcp add --transport http datadog https://mcp.datadoghq.com/v1/mcp
# then run /mcp inside a Claude Code session to complete Datadog's OAuth sign-in
claude mcp list

The default local scope stores these entries in your user configuration (~/.claude.json), not in the repository.

Run in your terminal. Codex uses --env and --url where Claude Code uses -e and --transport http:

codex mcp add kubernetes -- npx -y kubernetes-mcp-server@latest --config ~/.config/k8s-mcp.toml
codex mcp add postgres --env DATABASE_URI="$PERF_DB_URI" \
  -- uvx postgres-mcp --access-mode=restricted
codex mcp add grafana --env GRAFANA_URL=https://grafana.example.com \
  --env GRAFANA_SERVICE_ACCOUNT_TOKEN="$GRAFANA_RO_TOKEN" -- uvx mcp-grafana --disable-write
# Datadog instead of Grafana (OAuth, so log in after adding it):
codex mcp add datadog --url https://mcp.datadoghq.com/v1/mcp
codex mcp login datadog
codex mcp list

Add the servers to your user-level ~/.cursor/mcp.json, which is not in the repository, then confirm each one shows as connected in Cursor’s MCP settings. The ${env:NAME} values are read from your environment when Cursor starts the server. If a server fails to start, check in Cursor’s MCP settings that the variable was resolved:

{
  "mcpServers": {
    "kubernetes": {
      "command": "npx",
      "args": ["-y", "kubernetes-mcp-server@latest", "--config", "/home/you/.config/k8s-mcp.toml"]
    },
    "postgres": {
      "command": "uvx",
      "args": ["postgres-mcp", "--access-mode=restricted"],
      "env": { "DATABASE_URI": "${env:PERF_DB_URI}" }
    },
    "grafana": {
      "command": "uvx",
      "args": ["mcp-grafana", "--disable-write"],
      "env": {
        "GRAFANA_URL": "https://grafana.example.com",
        "GRAFANA_SERVICE_ACCOUNT_TOKEN": "${env:GRAFANA_RO_TOKEN}"
      }
    }
  }
}

For Datadog instead of Grafana, add a remote entry: "datadog": { "url": "https://mcp.datadoghq.com/v1/mcp" }.

All three setups take the database URI and the Grafana token from PERF_DB_URI and GRAFANA_RO_TOKEN. Load them into your shell from your secret store (for example export PERF_DB_URI="$(vault kv get -field=uri secret/perf-ro-db)") rather than typing the values. In Cursor, ${env:NAME} is resolved when the server starts. With claude mcp add and codex mcp add, your shell substitutes the value when you run the command, and the resolved secret is stored in plaintext in ~/.claude.json or ~/.codex/config.toml. Keep those files readable only by you (chmod 600), use a short-lived read-only credential, and rotate it after the investigation. Replace the Grafana host with your own. get_top_queries in Postgres MCP Pro needs the pg_stat_statements extension, and its hypothetical-index analysis needs hypopg; most managed Postgres services offer both.

How do you diagnose a p99 regression without changing anything?

Start the session in a read-only planning mode, then ask for a diagnosis only. The “no fixes yet” rule stops the agent from pattern-matching to a plausible remedy before it has read your data.

Plan mode keeps Claude Code to research and a written plan, which matters because interactive sessions otherwise start in auto mode (v2.1.283 or later, both release channels on 2026-10-12):

claude --permission-mode plan

Paste the prompt below at the session prompt. Claude Code asks for approval before any MCP call your settings do not already allow.

The built-in :read-only permission profile (beta) keeps the model’s shell commands from writing:

codex -c default_permissions=":read-only"

Paste the prompt below into the session.

Use the kubernetes, grafana (or datadog) and postgres MCP servers. Do not
change anything and do not propose fixes yet.

checkout-service p99 went from ~200 ms to ~3 s starting around 18:00 UTC
today. Compare 17:00-18:00 UTC (baseline) with 18:00-19:30 UTC (incident):

1. Pod CPU/memory, restarts and replica count for checkout-service.
2. The downstream span or dependency whose latency grew the most.
3. Top queries by total time from pg_stat_statements, and connection
   counts against max_connections.
4. Deploys, config changes or HPA events in the 30 minutes before 18:00.

Rank the candidate causes. For each, cite the exact metric, query or event,
its value, and its timestamp, so I can check it in the dashboard. Label
anything you inferred rather than read as INFERRED.

Before you act, open the dashboard and check two of the cited numbers yourself. If both match, the diagnosis rests on live data; if either does not, see when agent-assisted tuning goes wrong.

How do you size connection pools against Postgres limits?

A common peak-hour failure is sorry, too many clients already from Postgres while the database CPU is idle: the database is not slow, it is out of connection slots. The arithmetic is mechanical, which suits an agent, but it is easy to get wrong in one specific way. Current replica counts understate demand; the HPA’s maxReplicas is what you reach at peak.

Measure demand at the scaling ceiling. Ask the agent for Σ(pool max × HPA maxReplicas) across every service that shares the instance, against max_connections minus superuser_reserved_connections and any connections held by migrations, cron jobs, and admin tools.
Get a concrete configuration, not advice. Have the agent propose per-service pool sizes and, if the sum still overflows, a PgBouncer layer in pool_mode = transaction with explicit max_client_conn and default_pool_size.
Check what transaction pooling breaks. Transaction pooling does not preserve session state between transactions: SET statements, session advisory locks, and LISTEN behave differently, and protocol-level prepared statements break unless PgBouncer’s max_prepared_statements is enabled (PgBouncer 1.21 and later; check its value in your PgBouncer version, because it was off by default when it arrived in 1.21). Many drivers and ORMs prepare statements by default, so check your driver’s prepared-statement setting, and ask the agent to search the code for the rest before you switch.
Roll out to one service and watch a real peak. Change the pool settings in one pull request, and record the previous values in its description, so rollback is either a single revert or a copy of the old values if the setting lives outside the repository.

Use the kubernetes and postgres MCP servers, read-only. Eight services share
one Postgres instance and we see "too many clients already" at peak.

1. For every Deployment that connects to this database, read the pool
   min/max from its env or ConfigMap, its current replicas, and its HPA
   maxReplicas.
2. Read max_connections, superuser_reserved_connections, and the current
   connections grouped by application_name.
3. Show a table: service, pool max, current replicas, maxReplicas, demand at
   maxReplicas. Sum it and compare with the usable limit.
4. Propose per-service pool sizes that fit with 20% headroom. If they cannot
   fit, propose PgBouncer in transaction mode with max_client_conn and
   default_pool_size, and grep this repository for SET, advisory locks,
   LISTEN and protocol-level prepared statements (check the driver/ORM
   prepared-statement setting) that transaction pooling would break.

Show the arithmetic. Do not apply anything.

How do you stop an autoscaler from flapping?

An HPA that scales up late and then scales down every two or three minutes costs money and still misses the spike. Two Kubernetes behaviours explain most cases, and the agent should check both before proposing numbers:

Scale-down stabilization. behavior.scaleDown.stabilizationWindowSeconds defaults to 300 seconds. Scale-down every few minutes usually means someone shortened it.
Utilization is relative to requests. A CPU target of 70% means 70% of the container’s CPU request. A request set far below real usage makes the HPA react to noise; one set far above makes it react late.

Use the kubernetes and grafana (or datadog) MCP servers, read-only.

payment-service HPA targets 70% CPU (min 3, max 10). CPU reaches 95% before
it scales up, and it scales down every 2-3 minutes.

1. Read the HPA spec including any behavior block, and the container CPU
   requests and limits.
2. From the last 7 days of metrics, compare actual CPU usage per pod with the
   request, and plot request rate against replica count at the daily peaks.
3. Explain why scaling is late and why it flaps, citing the fields and values.
4. Output a patched HPA manifest with explicit behavior.scaleUp and
   behavior.scaleDown policies and stabilizationWindowSeconds, and say whether
   a request-rate metric would lead CPU for this service. One line per
   changed field explaining why.

Write the manifest to a file for review. Do not apply it.

Review the manifest like any other infrastructure change: it goes through a pull request and your normal infrastructure-as-code workflow, not a kubectl apply from the agent session.

How do you prove a performance change worked?

“The agent says it’s faster” is not evidence. Write the acceptance criteria before the change and let measurements decide. Nobody has to read the agent’s reasoning to sign off, because the numbers either meet the criteria or they do not.

Write the acceptance criteria first. One metric, one threshold, one window: “checkout p99 under 400 ms and error rate under 0.5% across the 18:00–21:00 UTC peak, with Postgres connections below 80% of max_connections”. Put them in the pull request.
Load-test in staging before merge. Replay a production-shaped load with the tool your team already uses (see load, stress, and benchmark testing). A pool or HPA change that only passes a happy-path test proves nothing about peak concurrency.
Canary in production. Ship to one service or a small share of replicas first. If you already run progressive delivery, gate promotion on the same metric as the acceptance criteria.
Compare the same window. After the next real peak, compare it with the same weekday a week earlier, because a Friday peak and a Wednesday peak have different traffic shapes. Have the agent write the verification report with the prompt below, then check two of its numbers in the dashboard.
Sign off or roll back. The service owner signs off against the criteria. If a criterion fails, revert: kubectl rollout undo deployment/checkout-service for a Deployment change, or revert the pull request for HPA and pool configuration.

Use the grafana (or datadog), kubernetes and postgres MCP servers, read-only.

The change in PR #1482 (checkout-service pool size and PgBouncer) reached
100% of replicas at 14:00 UTC on 2026-09-24. Acceptance criteria:
p99 < 400 ms, error rate < 0.5% over 18:00-21:00 UTC, Postgres connections
< 80% of max_connections.

Compare 18:00-21:00 UTC on 2026-09-18 (before) with 18:00-21:00 UTC on
2026-09-25 (after). Also compare request volume, so a quieter day is not
mistaken for an improvement. Both dates are Fridays, so the traffic shape
matches.
Output a table: criterion, before, after, threshold, PASS/FAIL, and the exact
query you used for each row.

Attach the report and the queries to the change record. On teams that ship agent-written changes with an evidence bundle, this table is the performance section of it.

Which model should you use for performance diagnosis?

Start on the tool’s default model: Claude Opus 5.5 in Claude Code (on both release channels) and GPT-6.1 Sol in Codex. Raise the effort level before you switch model, and switch only when your own comparisons show a gain. Correlating traces, metrics, and query plans is long-context work, so the cheaper tiers suit single-signal checks such as “list the top ten queries”. Prices and the full lineup are on the AI models hub.

When agent-assisted performance tuning goes wrong

The agent answers without data. Round numbers (“around 95% CPU”) and no timestamps mean it is not reading your systems. Run claude mcp list or codex mcp list, or check Cursor’s MCP settings; fix the failing server and re-run the prompt. Do not accept a diagnosis from a session where a server was down.
A cited number does not match the dashboard. The model filled a gap in a query result. Ask it to print the exact query and the raw result for that number, and re-run the query yourself. Treat everything else from that answer as unconfirmed.
The fix treats a symptom. Latency did not move after the change because the pool or HPA was downstream of the real cause. Re-run the diagnosis prompt with the new data and ask it to re-rank the causes, excluding the one you already fixed.
Staging passed, production regressed. A pool that is fine at staging concurrency starves at peak. Roll back first, then load-test at the concurrency the HPA’s maxReplicas implies, not the average.
The Postgres server returns nothing useful. get_top_queries needs pg_stat_statements. Also note that the last PyPI release of postgres-mcp is 0.3.0 from 2025-05-16 (checked 2026-09-26); if a newer Postgres feature fails through it, fall back to running EXPLAIN (ANALYZE, BUFFERS) on a replica yourself.
The agent got write access by accident. A kubeconfig with the cluster-admin context, or --access-mode=unrestricted copied from a README, turns a diagnosis into a risk. Rotate to the read-only credentials above and check the audit log for anything the session changed.

Where to go next with production performance

AI-powered incident response: when the regression is an outage, run the incident workflow first and tune afterwards.
Cloud cost management and FinOps: right-sizing requests and autoscaling is also a cost question.
MCP security: how to scope credentials and review MCP servers before they reach production data.
Database development with AI: turning the slow queries you found into indexed, tested schema changes.