API Test Automation

API test automation with AI agents means generating endpoint tests from the API contract and acceptance criteria, not from the handler code, then proving the suite can fail. The core set is a status-and-schema suite per operation, a role-by-resource authorization matrix and, where several consumers deploy independently, consumer-driven contracts with Pact checked by can-i-deploy.

This page is for developers who ask an agent for “tests for every endpoint”, and for tech leads who have to decide whether those tests are allowed to gate a merge. The typical failure: the agent reads src/routes/orders.ts, writes 60 green tests, and every one of them asserts what the handler already does, including the missing ownership check that lets user A read user B’s orders.

What you’ll walk away with from API test automation

A test-plan prompt that turns openapi.yaml and the acceptance criteria into a reviewable table before any test code exists.
A generation prompt for one operation that covers success, validation, business rules, auth and edge cases.
An authorization-matrix pattern for object-level authorization checks, which a handler and a test written from that handler can both omit.
A Pact setup with a can-i-deploy gate, plus an MCP server that lets the agent query the broker.
Four checks that tell you whether the generated suite catches bugs, without reading it line by line.

Which API tests belong on this page, and which gates live elsewhere?

API testing overlaps with three neighbours. Keep each concern in one place so the agent does not write the same check three times.

Layer	Question it answers	Where it lives
Contract gates	Is the contract valid, backward compatible, and does the server match it under fuzzing?	Spectral, `oasdiff`, Schemathesis on API development
Endpoint behaviour	Does each operation enforce the rules in the spec: statuses, validation, business rules, who may do what?	This page
Consumer contracts	Will this provider version break a consumer that is already in production?	This page (Pact)
Service stack	Do the database, queue and third-party calls behave together?	Integration testing
Attack surface	Injection, SSRF, secrets, dependency CVEs	Security testing

The contract gates prove the code matches the contract. They cannot prove the contract matches the product: a schema says total is a number, not that it includes tax. That is what endpoint tests written from the acceptance criteria are for.

Generate API tests from the contract, not from the handlers

An agent that reads the handler before it writes the test copies the handler’s assumptions into the assertion. Give it the contract and the acceptance criteria as the oracle, and let it read the handler only to learn how to call it. For writing criteria an agent cannot misread, see acceptance criteria.

Ask for a test plan, not tests. The agent produces a table: operation, scenario, expected status, expected body rule, source line in the spec. This is the only artifact a human reads in full, and it takes minutes rather than an hour of reviewing test code.
Review the plan against the product. Add the rules the spec forgot (idempotency, ownership, limits) and delete scenarios that test the framework rather than your code. Commit the approved plan next to the tests, for example tests/api/plans/orders.md.
Generate the tests from the approved plan, one operation or resource per session. Smaller batches keep the agent’s context on one contract and make failures easier to attribute.
Make every test fail once. The agent breaks the behaviour a test covers (removes a validation, flips a status code), runs the suite, shows the red test, and reverts. A test that stays green against a broken handler is deleted or rewritten.
Lock the suite before the next feature session. Test paths become read-only for the implementing agent, and CI runs the suite from outside the diff. How to do that for each tool is on protecting the oracle.

Read api/openapi.yaml and docs/specs/orders.md. Do not open anything under
src/ yet.

For every operation under /orders, produce a markdown table with columns:
operation | scenario | request setup | expected status | expected body rule |
spec reference (file + heading or operationId).

Cover, per operation:
- the success case, with the documented response schema
- one row per required field missing, and one per documented constraint
  (min, max, enum, format)
- every business rule in docs/specs/orders.md (stock, limits, idempotency)
- 401 without a token, 401 with an expired token
- 403 or 404 when the caller is authenticated but does not own the resource
  (say which one the spec requires; if it does not say, list it under
  "Open questions")
- 404 for an unknown id, 409 or 422 where the spec defines a conflict

End with "Open questions": every behaviour the tests need but the spec does
not define. Do not invent answers. Save the result to
tests/api/plans/orders.md and stop.

The “Open questions” list is the most valuable output. Each entry is a product decision the agent would otherwise have made silently, and in the handler, not in the spec.

Generate the tests for one operation

Once the plan is approved, the generation prompt is the same in every tool. What differs is how you keep the agent from editing the handler to make its own tests pass.

Implement the approved plan in tests/api/plans/orders.md for POST /api/orders
as tests/api/orders.create.test.ts, using supertest against the app exported
from src/app.ts and the factories in tests/factories/. Follow the style of
tests/api/auth.api.test.ts.

Rules:
- Every row in the plan becomes one test whose name quotes the scenario.
- Assert the exact status code and validate every response body against the
  schema from api/openapi.yaml with additionalProperties treated as false.
- For 4xx responses, assert the error code field, not the message text.
- Each test seeds its own data inside a transaction that rolls back; no test
  depends on another test's rows.
- Do not modify anything under src/. If a test fails because the handler is
  wrong, leave the test red and list the failure with the plan row it proves.

Finish with: npm test -- tests/api/orders.create.test.ts

Then prove that each test can fail. This step deliberately breaks the handler, so run it in a session without the src/ restriction, or on a throwaway branch, never in the generation session.

For each test in tests/api/orders.create.test.ts, one at a time:
1. Break the behaviour it covers in src/ (remove the check, change the
   status code, drop the field).
2. Run only that test and confirm it fails.
3. Revert the change with git checkout -- src/ and confirm git status shows
   no changes under src/.

Do not edit any test in this step. Report every test that stayed green
while its behaviour was broken, with the change you made, so it can be
fixed or deleted.

Run the plan step with reads allowed and source writes denied, using the same --disallowedTools command shown below: the session can read the contract and save tests/api/plans/orders.md, and the prompt keeps it out of src/. Alternatively, use claude --permission-mode plan and save the plan yourself when Claude presents it, because plan mode blocks file writes. Start the generation session from the repository root with source edits denied, so the edit tools cannot touch src/:

claude --disallowedTools "Edit(src/**)" "Write(src/**)"

The deny rules cover only the Edit and Write tools (checked on 2.1.283). A Bash command such as sed -i or a shell redirect can still write to src/, and auto mode may approve it. So end the session with git diff --stat -- src/: if it lists anything, discard the run.

The seen-failing prompt needs to break the handler, so run it in a second session without that restriction, or on a throwaway branch. The per-tool setup for API work is on building APIs with Claude Code.

Put the test commands and the “do not modify src/ while writing tests” rule in AGENTS.md, which Codex reads before each task. Run the generation non-interactively from a clean branch with codex exec --sandbox workspace-write "<prompt>" and review the diff with git diff --stat first: any file outside tests/ is a reason to discard the run. The contract-first variant for Codex is on contract-first APIs in Codex.

Use Plan Mode for the test plan, then switch to the agent for generation with tests/api/plans/orders.md and api/openapi.yaml attached as context. Add a project rule that test-writing tasks must not edit src/, and check the changed-files list before you accept. On the pull request, Bugbot reviews for bugs, but it does not replace step 4: a reviewer cannot see a test that never fails.

How do you test API authorization so agents do not miss object-level checks?

Broken object-level authorization (BOLA) is an API bug that a handler and a test written from that handler can both omit. The handler checks that the caller is logged in, the test checks that a logged-in caller gets 200, and nobody checks that the order belongs to the caller. One 403 test per route does not catch it. A matrix does.

Define the matrix once as data: roles, the resources each role owns, and the expected outcome for each combination. The test iterates over every route in the contract, so a new route without a matrix entry fails the suite instead of passing unnoticed.

// tests/api/authz-matrix.test.ts (shape only; the agent fills in the rows)
const matrix = [
  { op: 'GET /orders/{id}',    actor: 'owner',       expect: 200 },
  { op: 'GET /orders/{id}',    actor: 'other-user',  expect: 404 },
  { op: 'GET /orders/{id}',    actor: 'support',     expect: 200 },
  { op: 'DELETE /orders/{id}', actor: 'support',     expect: 403 },
  { op: 'DELETE /orders/{id}', actor: 'anonymous',   expect: 401 },
] as const;
// A companion test loads api/openapi.yaml and fails when any operation
// has no row for every actor.

Build tests/api/authz-matrix.test.ts.

Actors: anonymous, owner (a user who created the resource), other-user
(a user in the same tenant who did not), other-tenant (a user in a different
tenant), support (role "support"), admin (role "admin").

1. Load every operation from api/openapi.yaml.
2. For each operation and each actor, look up the expected status in
   tests/api/authz-matrix.ts. Write that file as a typed table; take the
   expectations from docs/specs/permissions.md only. Where the spec is
   silent, put the row under a TODO block and do not guess.
3. Seed one resource per operation owned by "owner" in tenant A, call the
   operation as each actor, and assert the exact status. For 404-as-denial,
   also assert the body does not reveal that the resource exists.
4. Add a coverage test that fails if any operation in the contract lacks a
   row for any actor, and print the missing pairs.

Then remove the ownership check from the GET /orders/{id} handler, show the
matrix failing for other-user and other-tenant, and revert.

The TODO block goes to the product owner, not back to the agent. Whether support staff may cancel an order is a decision, and the matrix makes it visible.

When to use Pact contract tests instead of a schema diff

A breaking-change diff with oasdiff protects consumers from changes to the published contract. It does not know which fields a consumer actually reads, and it cannot tell you whether the provider version you are about to deploy works with the consumer versions already in production. Consumer-driven contracts answer that.

Situation	Enough	Add Pact
One team owns the API and its only client, deployed together	`oasdiff` in CI	No
Public API with unknown consumers	`oasdiff`, versioning policy	No, you cannot collect their pacts
Several internal consumers (web, mobile, other services) deploying on their own schedules	`oasdiff`	Yes
Mobile apps where old versions stay installed for months	`oasdiff`	Yes, with the old app versions recorded as deployed

Set up consumer-driven contract tests between web-app (consumer) and
order-service (provider) with @pact-foundation/pact.

Consumer side, in web-app:
- tests/contracts/order-service.pact.test.ts with interactions for
  GET /api/orders?page=1, GET /api/orders/{id}, POST /api/orders, and the
  400 and 404 error shapes.
- Use matchers (like, eachLike, regex for ids and ISO dates) so the pact
  pins the fields web-app reads, not example values. List every field the
  web-app source actually reads and include only those.
- Publish the pact from CI with the git SHA as the consumer version and the
  branch name as the branch.

Provider side, in order-service:
- tests/contracts/provider.verify.test.ts that fetches pacts from the broker
  for the main branch and deployed versions, and sets up each provider state
  ("order 42 exists for user u1") by seeding the test database.
- Publish verification results only from CI, never from a laptop.

Read PACT_BROKER_BASE_URL and the broker credentials from the environment:
PACT_BROKER_TOKEN for PactFlow, or PACT_BROKER_USERNAME and
PACT_BROKER_PASSWORD for a self-hosted Pact Broker. Do not write any of these
values into a file.

Before each deploy, CI asks the broker whether this version is compatible with everything already in the target environment. The command exits non-zero when a verification is missing or failed, so it works as a gate on its own:

# CI, before the deploy step. PACT_BROKER_BASE_URL plus PACT_BROKER_TOKEN (PactFlow)
# or PACT_BROKER_USERNAME/PACT_BROKER_PASSWORD (self-hosted broker) come from the CI secret store.
npx -p @pact-foundation/pact-cli pact-broker can-i-deploy \
  --pacticipant order-service --version "$GIT_SHA" \
  --to-environment production \
  --retry-while-unknown 10 --retry-interval 30

After the deploy, record it with pact-broker record-deployment --pacticipant order-service --version "$GIT_SHA" --environment production so the next can-i-deploy checks against what is really running. Versions checked on 2026-09-26: @pact-foundation/pact 17.1.4 and @pact-foundation/pact-cli 18.1.2 on npm.

Let the agent query the Pact Broker through MCP

When a verification fails, the agent needs the failing interaction and the provider state, not a screenshot of the broker. SmartBear’s MCP server (@smartbear/mcp, 0.41.3 on npm on 2026-09-26) exposes the broker to the agent, with tools including Can I Deploy, Matrix, Get Provider States and Get Pacts for Verification. It also ships Generate Pact Tests and Review Pact Tests, which it gates on your PactFlow AI entitlement. It works with PactFlow and with a self-hosted Pact Broker, but the two authenticate differently: the server sends PACT_BROKER_TOKEN as a bearer token, which is what PactFlow expects, and sends PACT_BROKER_USERNAME with PACT_BROKER_PASSWORD as basic auth, which is what a self-hosted broker with authentication enabled expects. The configs below are for PactFlow; the self-hosted variant follows them. They pin that version, because the server runs with your broker credentials in its environment: bump it deliberately, after reading the release notes, rather than pulling @latest.

Add it to the project’s .mcp.json. Claude Code expands ${VAR} from your environment, so the token never lands in the file:

{
  "mcpServers": {
    "smartbear": {
      "command": "npx",
      "args": ["-y", "@smartbear/mcp@0.41.3"],
      "env": {
        "PACT_BROKER_BASE_URL": "${PACT_BROKER_BASE_URL}",
        "PACT_BROKER_TOKEN": "${PACT_BROKER_TOKEN}"
      }
    }
  }
}

Add it to ~/.codex/config.toml. env_vars forwards the named variables from your shell instead of storing their values (checked with codex mcp get on 0.157.1):

[mcp_servers.smartbear]
command = "npx"
args = ["-y", "@smartbear/mcp@0.41.3"]
env_vars = ["PACT_BROKER_BASE_URL", "PACT_BROKER_TOKEN"]

Add the same command and args entry under mcpServers in .cursor/mcp.json. How Cursor passes environment variables to a stdio server was not verified for this page (cursor.com was unreachable on 2026-09-26); check Cursor’s MCP documentation before you put a token in the file, and never commit one.

For a self-hosted Pact Broker, replace the token with the two basic-auth variables, otherwise the broker rejects the bearer header with 401. In .mcp.json the env block becomes:

"env": {
  "PACT_BROKER_BASE_URL": "${PACT_BROKER_BASE_URL}",
  "PACT_BROKER_USERNAME": "${PACT_BROKER_USERNAME}",
  "PACT_BROKER_PASSWORD": "${PACT_BROKER_PASSWORD}"
}

and in Codex env_vars = ["PACT_BROKER_BASE_URL", "PACT_BROKER_USERNAME", "PACT_BROKER_PASSWORD"]. Set only one scheme: when PACT_BROKER_TOKEN is present the server uses it and ignores the username and password (checked in the @smartbear/mcp 0.41.3 client source).

Give the agent read-only credentials: a read-only token on PactFlow, or the read-only basic-auth user on a self-hosted broker (configured on the broker side, for example PACT_BROKER_BASIC_AUTH_READ_ONLY_USERNAME in the pactfoundation/pact-broker image). Publishing pacts, recording deployments and changing pacticipants belong to CI.

What changes when you test a GraphQL API?

The planning and matrix patterns carry over unchanged; the schema replaces openapi.yaml as the contract. Three things differ. Most servers answer 200 even when a resolver fails, so assert errors[].extensions.code rather than the status; request-level errors (parse, validation, depth limits) may come back as 4xx, depending on the server. Authorization lives on fields and resolvers, so the matrix is per field, not per route. And one query can be arbitrarily expensive, so depth and complexity limits need their own tests. Schema-level breaking changes belong to graphql-inspector diff, covered on API development.

Write tests/api/graphql/ for the schema in schema.graphql, sending requests
with supertest to POST /graphql on the app from src/app.ts.

1. For each query and mutation, a success test that validates the data shape
   against the schema and checks that only requested fields are returned.
2. For each mutation, one test per input validation rule in
   docs/specs/*.md, asserting errors[0].extensions.code (never the message).
3. A field-level authorization matrix: for User.email, User.orders and
   Order.paymentMethod, query as owner, other-user and admin and assert
   which fields come back null with a FORBIDDEN error.
4. Limits: a query nested 12 levels deep and a query requesting
   users(first: 10000) must both be rejected with the codes the server
   defines; find them in src/graphql/limits.ts and cite the line.
5. If the schema has subscriptions, one test that orderStatusChanged fires
   for the owner and does not fire for other-user.

Break the ownership check on Order.paymentMethod, show the red test, revert.

How do you know the generated API tests are any good without reading them?

Reading 60 generated tests does not tell you whether they catch bugs; it tells you whether they look reasonable. Four checks give you evidence instead, and all of them run in CI.

Check	What it proves	How
Seen failing	Each test can go red	Step 4 above; keep the agent’s report in the pull request
Mutation score on handlers	The suite detects small changes to real logic	After `npm init stryker` (which installs a test-runner plugin and writes the config): `npx stryker run --mutate "src/routes/orders.ts"` (`@stryker-mutator/core` 10.0.0 on npm, 2026-09-26)
Operation coverage	Every operation and documented status has a test	A test that loads `api/openapi.yaml` and fails on any operation-status pair without a test, like the matrix coverage test
Strict schema validation	Responses carry no undocumented fields	Validate with `additionalProperties: false` semantics in every test

Line coverage is the wrong measure here: a handler can be 100% covered by tests that assert nothing about ownership. Mutation score measures what the assertions notice. How to set a threshold and read the report is on oracle strength.

Sign-off splits cleanly. A person approves the contract, the test plan and the authorization matrix, because those encode product decisions. CI owns everything after that: the suite, the mutation threshold, can-i-deploy, and the contract gates. The pull request carries the plan, the seen-failing report and the mutation score as its evidence bundle, so the reviewer checks evidence rather than lines.

When agent-written API tests break down

The tests pass, and the bug they should catch is in production. The agent derived expectations from the handler. Recovery: regenerate from the plan with src/ hidden during planning, then run the seen-failing step on the missed behaviour. Add the case to the plan so the next session keeps it.

The agent loosened an assertion to get green. Typical signs are expect([200, 500]).toContain(res.status), toBeDefined() on a body, or a deleted test. Recovery: block edits to existing test files for implementing sessions, and have CI flag any diff that reduces assertions or test count in tests/api/. The detectors are on protecting the oracle.

Tests are flaky because they share database state. Parallel test files see each other’s rows, and order-dependent tests pass locally and fail in CI. Recovery: one transaction per test that rolls back, or a fresh schema per worker; ask the agent to run the suite three times in random order before it reports done. Fixture patterns are on test data management.

Every response is snapshotted and every change breaks 40 tests. Whole-body snapshots test field order and timestamps, not behaviour. Recovery: replace snapshots with schema validation plus explicit assertions on the fields the scenario is about.

A Pact verification fails after an intentional provider change. That is the gate working. Recovery: ship the change behind an additive field or a new version, let consumers update their pacts, and deploy the provider only when can-i-deploy passes. Do not ask the agent to “make the verification pass”; it will edit the provider state or the matcher.

Provider states drift from the pacts. A consumer adds “order 42 is cancelled” and the provider has no such state, so verification fails with a missing-state error. Recovery: give the agent the failing state name through the broker MCP server and ask it to add only that state’s setup in the provider test.

Tests pass in CI and the API behaves differently in production. Rate limits, caching, feature flags or middleware differ between environments. Recovery: list the production-only middleware in the test plan’s “Open questions”, and cover the rest with a post-deploy smoke run and progressive delivery.

Where to go next with API testing

API development Contract lint, breaking-change diffs and Schemathesis fuzzing as CI gates.

Oracle strength Mutation score and the bar a suite must clear before it gates a merge.

Integration testing Test the database, queues and external calls behind the API.

Security testing Injection, secrets and dependency scanning for the same endpoints.