End-to-End Test Automation

An agent-maintained end-to-end (E2E) suite is a set of Playwright tests that a coding agent writes from acceptance criteria and keeps passing as the UI changes, while CI proves the suite still catches regressions. The qualifier that matters: the agent writes the tests, but humans own the list of journeys, the visual baselines, and any change that weakens an assertion.

This page is for the developer who asks an agent to “add E2E tests” and the tech lead who has to trust the result. The agent produces 40 specs, CI is green, and a week later checkout breaks in production. Nobody read the specs, so nobody noticed that half of them assert a toast instead of a saved order, and three carry a waitForTimeout(5000) that hides a race.

What an agent-maintained E2E suite gives you

• A journey budget that decides which flows get an E2E test and which belong in cheaper layers
• A playwright.config.ts that makes flaky tests, stray test.only calls and silent snapshot updates fail in CI
• A browser connection for Claude Code, Codex, and Cursor, so the agent writes locators against the real page
• Copy-paste prompts that generate tests from acceptance criteria, refactor them into fixtures, set up visual checks, and fix flakiness without weakening assertions
• Four checks that show the suite catches bugs, none of which require reading the specs line by line

The examples use Playwright Test 1.63.0 (npm latest on 2026-09-26). On Cypress the journey budget, the oracle checks and the prompt rules (persisted-outcome assertions, no sleeps, no skips) still apply; the commands, locator APIs and config keys do not.

Which user journeys deserve an E2E test?

E2E tests are the slowest and most expensive layer, and agents write them faster than anyone can maintain them. Decide the budget before the agent writes anything. A tech lead owns this list, because it is the part of the oracle that no agent should extend or shrink on its own.

Put it in E2E when…	Put it in a cheaper layer when…
The flow crosses the browser, the API, and storage, and a user would notice a break (sign-up, checkout, publish)	The rule is pure logic, such as price rounding or validation: unit tests
The bug class lives in the browser: autofill, focus, iframes, client routing, hydration	The contract is between services: API tests or integration tests
A regression would cost revenue or trust before monitoring caught it	The check is a one-off proof for one change: agent-driven browser verification

Keep the list in the repository, next to the suite, as tests/e2e/JOURNEYS.md: one line per journey, the acceptance criterion it proves, and the spec file that proves it. Reviewers read that file instead of the specs. The criteria themselves come from acceptance criteria an agent can verify.

Set up the suite before the agent writes a test

An agent copies whatever the existing suite does. Put the rules in the config and the fixtures first, so the first generated test inherits them.

1. Install Playwright Test and one browser in your terminal:

   npm init playwright@latest -- --quiet --browser=chromium --install-deps

2. Replace the generated playwright.config.ts with a config that turns silent failures into loud ones:

   playwright.config.ts

   import { defineConfig, devices } from '@playwright/test';
   
   const CI = !!process.env.CI;
   const baseURL = process.env.E2E_BASE_URL ?? 'http://localhost:3000';
   const port = new URL(baseURL).port || '3000';  // the dev server listens where the tests look
   
   export default defineConfig({
     testDir: './tests/e2e',
     fullyParallel: true,
     forbidOnly: CI,                               // a stray test.only fails the run
     retries: CI ? 1 : 0,                          // one retry so Playwright can label a test flaky…
     failOnFlakyTests: CI,                         // …and the flaky label fails the job
     updateSnapshots: CI ? 'none' : 'missing',     // CI never writes a baseline
     reporter: CI ? 'blob' : 'list',
     use: { baseURL, trace: 'retain-on-failure' },
     projects: [
       { name: 'setup', testMatch: /auth\.setup\.ts/ },
       {
         name: 'chromium',
         use: { ...devices['Desktop Chrome'], storageState: 'tests/e2e/.auth/user.json' },
         dependencies: ['setup'],
       },
     ],
     webServer: {
       command: `npm run dev -- --port ${port}`,  // assumes your dev script accepts --port
       url: baseURL,
       reuseExistingServer: !CI,
     },
   });

   updateSnapshots defaults to 'missing', which writes a new baseline whenever none exists. Setting it to 'none' in CI means a screenshot test without an approved baseline fails instead of approving itself.

3. Log in once through the API, not the UI, and save the session for every test:

   tests/e2e/auth.setup.ts

   import { test as setup, expect } from '@playwright/test';
   
   setup('authenticate', async ({ request }) => {
     const res = await request.post('/api/auth/login', {
       data: { email: process.env.E2E_USER_EMAIL, password: process.env.E2E_USER_PASSWORD },
     });
     expect(res.ok()).toBeTruthy();
     await request.storageState({ path: 'tests/e2e/.auth/user.json' });
   });

   Add tests/e2e/.auth/ to .gitignore. Point /api/auth/login at your own login endpoint, and use a disposable test account, never a real user.

4. Protect the oracle: add tests/e2e/, playwright.config.ts, and the snapshot folders to CODEOWNERS, so a human approves every change to them. Protecting the oracle covers deny rules that stop the agent from editing them at all.

Connect the agent to a browser

The agent writes better locators when it can open the page and read its accessibility tree than when it guesses from JSX. Playwright ships two routes: the Playwright MCP server, and the Playwright CLI with skills, which its README suggests coding agents might benefit from, because it avoids loading large tool schemas and accessibility snapshots into context. Playwright MCP vs Playwright CLI compares them in depth. The setup differs per tool:

Claude Code

# Browser tools for this project; --isolated keeps parallel agents from sharing a profile
claude mcp add playwright -- npx @playwright/mcp@latest --isolated

# Optional: Playwright's planner, generator and healer subagents
npx playwright init-agents --loop=claude

init-agents --loop=claude writes the three subagents to .claude/agents/ and a playwright-test server entry to .mcp.json. It overwrites an existing .mcp.json (checked in 1.63.0), so commit or merge your current entries first.

Codex

codex mcp add playwright -- npx @playwright/mcp@latest --isolated

# Optional: the same three test agents, written to .codex/agents/
npx playwright init-agents --loop=codex

Cursor

Add the server to .cursor/mcp.json:

{
  "mcpServers": {
    "playwright": { "command": "npx", "args": ["@playwright/mcp@latest", "--isolated"] }
  }
}

init-agents 1.63.0 has no Cursor loop (its choices are claude, codex, copilot, opencode, vscode, and vscode-legacy), so use the MCP server with the prompts on this page.

To use the CLI route instead of MCP in any of the three tools, run npm install -g @playwright/cli@latest (the unscoped playwright-cli package is deprecated), then install the skill where your tool reads it. playwright-cli install --skills defaults to claude and writes .claude/skills/playwright-cli/ for Claude Code; Codex reads project skills from .agents/skills/, so run playwright-cli install --skills=agents there (both checked in @playwright/cli 0.1.22). Cursor loads skills from .agents/skills/ and .cursor/skills/, and for compatibility also from .claude/skills/ and .codex/skills/ (Cursor docs, checked 2026-10-12), so both installs above are visible to it; confirm Cursor lists the skill before you rely on it. For a skill-based browser that is not tied to Playwright, see agent-browser.

Generate tests from acceptance criteria

Give the agent one journey at a time, with its criteria, and make it run what it writes. The prompt below works unchanged in Claude Code, Codex, and Cursor.

Copy-paste prompt: write E2E tests for one journey

Write Playwright tests for the journey "a signed-in user upgrades to the Pro plan"
from tests/e2e/JOURNEYS.md and its acceptance criteria in docs/specs/billing.md.

Before writing code, open the running app with the Playwright browser tools, walk the
journey once, and list the locators you will use.

Rules:
- one test per acceptance criterion, in tests/e2e/billing/upgrade.spec.ts
- locators: getByRole, getByLabel or getByTestId only; no CSS classes, no XPath, no nth()
- assert the persisted outcome: reload the page or open the account page and check the
  plan there; a toast alone is not evidence
- no waitForTimeout; use web-first assertions (await expect(...).toBeVisible())
- reuse the storageState from tests/e2e/auth.setup.ts; never log in through the UI
- use the payment provider's test mode; never call a live payment endpoint

Then run: npx playwright test tests/e2e/billing/upgrade.spec.ts --repeat-each=5 --retries=0
Report the pass count and add the spec to JOURNEYS.md. If a run fails, fix the test or
report the product bug. Do not add retries, sleeps, test.skip or test.fixme to get green.

--repeat-each=5 --retries=0 is the cheap flake filter: a test that fails once in five local runs fails far more often across a CI fleet. Ask for the command output, not a summary, so the pass count is evidence rather than a claim.

Page objects or fixtures: how to keep agent-written tests maintainable

Agents duplicate helpers. After ten journeys you usually have four login helpers and three ways to open the pricing page. Consolidate once the suite has a few journeys, not before, and keep the assertions visible in the spec file, where a reviewer can find them.

Copy-paste prompt: refactor specs into fixtures without changing what they check

Refactor tests/e2e/ so that navigation and form filling live in page objects under
tests/e2e/pages/, exposed through a custom fixture in tests/e2e/fixtures.ts
(test.extend). Keep every expect() call in the spec files, not in the page objects.

Before you start, record: the output of `npx playwright test --list` and the count of
expect( calls under tests/e2e (grep -rc "expect(" tests/e2e).
After the refactor, both must be identical: same test titles, same number of expect calls.
Run the whole suite with --retries=0 and report both before-and-after numbers.

The before-and-after numbers make the refactor verifiable without reading it: the same test titles and the same number of assertions mean the agent moved code without dropping checks. Page objects that hold assertions hide them from review; keeping expect in the spec is the rule that makes this check possible.

Add visual regression checks the agent cannot approve

toHaveScreenshot() compares a page against a stored baseline and disables CSS animations by default. The risk with an agent is not the comparison but the approval: npx playwright test --update-snapshots rewrites every baseline that no longer matches, including those that differ because of a bug (the flag’s preset is changed; --update-snapshots=all rewrites them all).

Copy-paste prompt: visual checks for the critical pages

Add visual regression checks for /pricing and /account in tests/e2e/visual.spec.ts.
- use await expect(page).toHaveScreenshot('<name>.png', { maxDiffPixelRatio: 0.01 })
- mask dynamic content with the mask option: dates, avatars, order numbers
- wait for web fonts with await page.evaluate(() => document.fonts.ready) before each shot
- test two viewports: Desktop Chrome and devices['Pixel 7']
Generate the first baselines once with the Docker image of the Playwright version in
package.json, so they match CI rendering. Never run --update-snapshots after that:
if a screenshot differs, stop and show me the diff image from test-results/.

Baselines are named per browser and operating system, so a baseline made on a Mac fails on a Linux CI runner. Generate them in the same container image CI uses (the workflow below runs in mcr.microsoft.com/playwright:v1.63.0-noble), once, from the repository root. The container needs its own npm ci, because the host’s node_modules may hold native binaries for another OS, and the test credentials, because every test depends on the login in auth.setup.ts:

docker run --rm --ipc=host -e E2E_USER_EMAIL -e E2E_USER_PASSWORD \
  -v "$PWD":/work -w /work mcr.microsoft.com/playwright:v1.63.0-noble \
  /bin/bash -c 'npm ci && npx playwright test visual.spec.ts --update-snapshots=missing'

On a Linux host the container writes the baselines as root, so add --user $(id -u):$(id -g) or run chown on the output afterwards. After that, a human looks at the diff image in the HTML report and approves each baseline change in the pull request; CODEOWNERS on the snapshot folder enforces it.

Run the suite in CI with sharding and a flake gate

Shard the suite across runners and merge the blob reports into one HTML report. The workflow runs on pull_request, so it runs the contributor’s code without write permissions. Forked pull requests receive no secrets, so auth.setup.ts would post empty credentials and every fork would fail E2E for a reason unrelated to the product; the if: below skips the job for forks. If you accept outside contributions, add a separate job that runs a no-auth subset for them instead. The job runs inside the Playwright image, which already contains the browsers, so there is no playwright install step.

.github/workflows/e2e.yml

name: e2e
on: pull_request
permissions:
  contents: read
jobs:
  e2e:
    # Forked pull requests get no secrets, so auth.setup.ts would fail; skip them
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.63.0-noble   # same image that made the baselines
      options: --user 1001
    strategy:
      fail-fast: false
      matrix:
        shard: [1/4, 2/4, 3/4, 4/4]
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright test --shard=${{ matrix.shard }}
        env:
          E2E_USER_EMAIL: ${{ secrets.E2E_USER_EMAIL }}
          E2E_USER_PASSWORD: ${{ secrets.E2E_USER_PASSWORD }}
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ strategy.job-index }}
          path: blob-report
          retention-days: 7
  report:
    if: ${{ !cancelled() && github.event.pull_request.head.repo.full_name == github.repository }}
    needs: e2e
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - uses: actions/download-artifact@v8
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
      - uses: actions/upload-artifact@v7
        with:
          name: playwright-report
          path: playwright-report
          retention-days: 14

The traces from failed tests (trace: 'retain-on-failure') land in the merged report. Link that artifact from the pull request’s evidence bundle, so a reviewer opens the failing step instead of rerunning it. Short-lived per-branch environments are covered in ephemeral environments.

How do you know the suite catches regressions without reading it?

A green suite proves only that the tests passed. Four checks prove the suite would fail when the product breaks, and none of them requires reading the specs.

1. Every new test goes red on the old code. Run the new spec against the base branch in a separate worktree; it must fail. Run npm ci in that worktree first, and give it its own port through E2E_BASE_URL with nothing else listening on it; the config above starts the dev server on that port. Otherwise reuseExistingServer attaches to the dev server of your main checkout, and the spec tests the new code instead of the old. Agent-driven browser verification has the prompt for this red-then-green proof.

2. Planted bugs get caught. Break each critical journey on purpose, one bug at a time, and count how many the suite detects. The prompt below runs this as a throwaway exercise; oracle strength turns it into a tracked mutation score.

3. Flakiness fails the build. failOnFlakyTests with one retry means a test that passes only on retry turns the job red instead of hiding.

4. Weakening is flagged. A CI step fails when a pull request removes expect lines or adds a skip or a sleep under tests/e2e/:

   # Steps in a job of a workflow that runs on pull_request
   - uses: actions/checkout@v7
     with: { fetch-depth: 0, persist-credentials: false }                # the diff needs the base branch's history
   - name: Flag changes that weaken the E2E oracle
     env: { BASE: '${{ github.base_ref }}' } # main, master, develop: whatever the PR targets
     run: |
       diff=$(git diff "origin/$BASE...HEAD" -- tests/e2e)
       removed=$(printf '%s\n' "$diff" | grep -cE '^-.*\bexpect\(' || true)
       added=$(printf '%s\n' "$diff" | grep -cE '^\+.*(test\.(skip|fixme)|waitForTimeout)' || true)
       if [ "$removed" -gt 0 ] || [ "$added" -gt 0 ]; then
         echo "E2E oracle weakened: $removed expect lines removed, $added skip/fixme/sleep lines added"
         exit 1
       fi

   Moving an assertion also trips it. That is deliberate: the move gets a human look. Make this a separate, non-required check, and treat a red result as needing approval from the CODEOWNERS owner of tests/e2e/.

Copy-paste prompt: plant bugs and count what the suite catches

Measure whether tests/e2e catches real regressions. Work in a throwaway worktree:
  git worktree add ../e2e-mutants HEAD
  cd ../e2e-mutants && npm ci
Give the worktree its own port, or the spec attaches to the dev server of the main
checkout and every bug reads as missed. Before the first run, confirm nothing listens
on port 3100 (lsof -i :3100 prints nothing), check that playwright.config.ts starts
the dev server on the port in E2E_BASE_URL, and run every spec as:
  E2E_BASE_URL=http://localhost:3100 npx playwright test <spec> --retries=0
For each journey in tests/e2e/JOURNEYS.md, make ONE small product-code change that a
user would notice (skip the save call, show the wrong plan name, drop a required field
check), run only that journey's spec with the command above, record caught or missed, then
revert the change before the next one. Never edit anything under tests/.
Report a table: journey, planted bug, caught or missed, failing assertion.
Finish with: git worktree remove --force ../e2e-mutants
For every missed bug, propose the assertion that would catch it; do not add it yet.

Sign-off stays with people and CI. The tech lead owns JOURNEYS.md and approves baseline and assertion changes; the CI job is the merge gate; the reviewer checks the journey list, the planted-bug table, and the report artifact instead of the spec lines.

Fix a flaky E2E test without weakening it

Agents have two fast ways to turn a flaky test green: add a sleep, or loosen the assertion. Both hide the bug the test exists to catch. Forbid them in the prompt and demand the evidence.

Copy-paste prompt: diagnose and fix a flaky test

The test "user filters products by category" in tests/e2e/catalog/filter.spec.ts
fails intermittently in CI. Diagnose it before changing anything.

1. Run it with --repeat-each=20 --retries=0 --trace=retain-on-failure and report the
   failure rate.
2. Open a failing trace (npx playwright show-trace <path>) and name the cause: an action
   before the element was ready, an assertion on data still loading, shared test data,
   a race between navigation and a request, or a product bug.
3. Fix the cause: web-first assertions, waiting for the specific response with
   page.waitForResponse, or isolated test data. If it is a product race, stop and report it.

Forbidden: waitForTimeout, raising timeouts, retries in the spec, test.skip or test.fixme,
removing or loosening any expect. Finish with 20/20 passes and show the command output.

When an agent-written E2E suite breaks

Failure	What you see	Recovery
Toast-only assertions	Tests pass while the saved data is wrong	Require a reload or a second page in every journey; run the planted-bug prompt to find the gaps
The agent approved a broken screenshot	A pull request updates .png baselines along with the UI change	Keep updateSnapshots: 'none' in CI and CODEOWNERS on snapshot folders; review the diff image, not the file list
Wrong dev server	The suite passes locally against another worktree’s server on the same port	Give each worktree its own port through E2E_BASE_URL and start its dev server on that port, as the config above does; reuseExistingServer attaches to whatever answers on url
Parallel agents collide in the browser	Browser launch fails, or one agent sees another’s session	Start Playwright MCP with --isolated or a separate --user-data-dir per worktree
Healer or fixer weakens a test	A red test turns green after an expect change or a new test.fixme()	The weakening check turns red, and CODEOWNERS on tests/e2e/ routes the pull request to its owner; revert the test change and fix the product or the locator
Suite too slow to run on every pull request	Developers and agents skip it	Shard it; tag the critical journeys (test('…', { tag: '@critical' }, …)) and run --grep @critical per pull request, the full suite nightly
Baselines differ on CI only	Font or anti-aliasing differences fail every screenshot	Generate and compare baselines in the same container image; mask text-heavy dynamic areas
Third-party iframe blocks the flow	Payment or CAPTCHA widgets fail in the test	Use the provider’s test mode and test keys, or stub the provider at the network layer with page.route

Where to go next with E2E testing

Agent-driven browser verification Prove one change in a real browser and promote the flow into a test.

Playwright MCP vs Playwright CLI Which browser route to give the agent, and what each costs in context.

How strong is your oracle? Mutation score and flake rate as the measure of a suite you did not read.

Protecting the oracle Stop agents from editing the tests that judge their work.

Accessibility testing Add automated WCAG checks to the same browser flows.

Test data management Seed isolated data so parallel E2E tests do not collide.