E2E policy: browser evidence for every user-visible change

An E2E policy for agent-built software sets the evidence each user-visible change must carry before merge. The agent runs the acceptance criteria in a real browser and retains screenshots, traces, and console and network logs. Critical flows then become deterministic CI tests that gate every later change. Browser runs add evidence; they never replace the deterministic gate.

This page is for the CTO who owns question 12 of the CTO Scorecard and for the tech lead who enforces it. The usual situation: agents now open most of the frontend pull requests, unit tests stay green, and the PR description says “verified in the browser”. Then a pricing page ships with a checkout button that does nothing on mobile, and nobody can show what was verified, on which account, at which viewport.

Q12 · Quality gates: What is the E2E requirement for UI changes?

Max-score answer: an acceptance-criteria browser run with retained evidence, and critical paths promoted to deterministic CI tests.

What this E2E policy gives you

• A scoring table for Q12 that names the one control that moves you up each level
• An evidence matrix by risk class that tells every agent and reviewer what a UI change must prove
• Browser setup for Claude Code, Codex, and Cursor with isolated, headless sessions that write evidence to disk
• A Playwright configuration and CI job that fail on flaky tests instead of retrying them into green
• A policy file you can commit as-is, and three copy-paste prompts that produce the evidence
• Self-tests and four monthly numbers that show the policy holds without anyone watching every run

This page covers the organizational rule. The developer’s hands-on loop is on agent-driven browser verification, and test-writing technique is on end-to-end test automation.

How do you score each level of Q12?

The scorecard awards 0–3 points. Each step adds a control, not a tool.

Points	What the scorecard sees	What moves you to the next level
0	No E2E requirement; UI changes are checked by hand, if at all	Pick the three flows that make or lose money and write one Playwright test for each
1	Playwright or Cypress recommended but not required	Make the critical-flow suite a required status check on the default branch
2	E2E required for new flows, with a manual gate	Require an acceptance-criteria browser run with retained evidence on every user-visible PR, and promote stable critical paths into CI
3	Criteria-driven browser runs with retained evidence; critical paths are deterministic CI tests	Keep it there with the self-tests and the monthly review below

What evidence does each class of UI change need?

Classify the change by what breaks when it is wrong, not by how many lines it touches. The class decides the evidence; the PR template asks for the class.

Risk class	Typical change	Agent browser run	Deterministic CI	Human sign-off
Low	Copy, static content, styling inside one component	Route loads; screenshot at the viewports your policy names; no new console errors	Existing suite stays green; link and automated accessibility checks	Reviewer looks at the screenshots
Medium	Component behavior, form validation, client state	One run per acceptance criterion, with trace	Component or integration test for the new behavior	Code owner
High	New user flow, navigation change, anything that writes data	Run per criterion, including negative cases	E2E happy path plus the main failure path, tagged @critical if it guards revenue or access	Code owner and product owner accept the evidence
Critical	Authentication, payment, destructive or privacy-affecting actions	Exploratory run on synthetic accounts only	E2E with negative cases, written or reviewed by a person, never edited by the agent in the same PR	Named approver per the gates and guardrails policy

Two rules hold in every class. The browser agent never touches production data. And an agent’s visual judgment (“the layout looks correct”) is evidence for a human, not a pass.

How do you roll the policy out?

1. Write acceptance criteria as observable behavior. Each criterion names a starting state, a role, a viewport, the actions, the expected UI, the backend effect, and one outcome that must not happen. The format is on executable acceptance criteria.

2. Give agents a controlled environment. Seeded test accounts, synthetic data, a local or ephemeral stack, and explicit cleanup. Production credentials are never in the agent’s environment; see ephemeral environments and test data management.

3. Install an isolated browser tool for every agent. Use a pinned version, a headless browser, an in-memory profile, and an output directory for evidence. The next section has the commands.

4. Retain evidence per criterion. The PR carries a table: criterion, result, and a path to the screenshot or trace, plus console and failed network requests. It goes into the evidence bundle that CI checks for completeness.

5. Promote critical paths. When a flow is stable and guards revenue, access, or data, the agent drafts a Playwright test tagged @critical, and a person reviews the assertions, not the selectors. Every reproduced UI regression gets a test in the same PR as its fix.

6. Make the critical suite a required check. Branch protection requires it on the default branch. On pull_request, CI runs the workflow, playwright.config.ts and the tests from the PR itself, so a PR can weaken its own gate. Put e2e/, playwright.config.ts and .github/workflows/ under CODEOWNERS and require code-owner review, so the agent cannot change @critical tests or the gate in the PR it is implementing without a person approving it; see protecting the test oracle.

Give every agent an isolated browser

The official Playwright MCP server (@playwright/mcp 0.0.82 on npm, checked 2026-09-26) works in all three tools. Its --isolated flag keeps the profile in memory, which parallel agents need because a persistent profile serves one browser at a time. --output-dir puts screenshots where the evidence table can link them. The flags below are from npx @playwright/mcp@0.0.82 --help.

Claude Code

Run in the repository root. --scope project writes .mcp.json, so the whole team gets the same pinned server through review.

claude mcp add --scope project playwright -- npx @playwright/mcp@0.0.82 \
  --isolated --headless --viewport-size 1280x720 --output-dir .evidence/browser

Claude Code also has a --chrome flag for the Claude in Chrome integration (claude --help, v2.1.283). It drives your own browser with your own sessions, so keep it for exploration and use the isolated server for policy evidence.

Codex

Run in a terminal. codex mcp add writes the server to ~/.codex/config.toml; commit a project copy only if your team manages Codex configuration that way.

codex mcp add playwright -- npx @playwright/mcp@0.0.82 \
  --isolated --headless --viewport-size 1280x720 --output-dir .evidence/browser

Codex 0.157.1 also ships built-in browser use and an in-app browser, both on by default (codex features list). They are useful for a quick look; for policy evidence, use the pinned server so every run writes files to the same place.

Cursor

Add the server to .cursor/mcp.json in the repository. This is the standard mcpServers configuration from the Playwright MCP README (0.0.82); Cursor also accepts the same command through Cursor Settings → MCP → Add new MCP Server.

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@0.0.82", "--isolated", "--headless",
               "--viewport-size", "1280x720", "--output-dir", ".evidence/browser"]
    }
  }
}

Add .evidence/ to .gitignore and upload it as a CI artifact instead. When MCP context cost matters, the agent-browser CLI with its skill (npm agent-browser 0.38.1) is the lighter alternative; the trade-offs are on agent-browser and browser automation MCP servers.

Configure the deterministic layer

The browser run shows the change works once. The CI suite proves it keeps working. Four settings make the suite trustworthy enough to gate merges. Every option below is in the @playwright/test 1.63.0 type definitions.

playwright.config.ts

import { defineConfig, devices } from '@playwright/test';

const baseURL = process.env.E2E_BASE_URL ?? 'http://localhost:3000';

export default defineConfig({
  testDir: './e2e',
  forbidOnly: !!process.env.CI,          // a stray test.only fails CI
  retries: process.env.CI ? 1 : 0,
  failOnFlakyTests: !!process.env.CI,    // pass-on-retry fails the run
  reporter: [['html', { open: 'never' }], ['list']],
  use: {
    baseURL,
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
  },
  projects: [
    { name: 'desktop', use: { ...devices['Desktop Chrome'] } },
    { name: 'mobile', use: { ...devices['Pixel 7'] } },
  ],
  webServer: {
    command: 'npm run start:test',
    url: baseURL,
    reuseExistingServer: !process.env.CI, // CI always starts its own app
  },
});

failOnFlakyTests is the policy setting. A retry still runs, so the trace shows what changed between attempts, but a test that fails and then passes fails the job instead of turning green. reuseExistingServer: !process.env.CI stops CI from attaching to an unrelated server that already holds the port and reporting green against code that was never under test.

.github/workflows/e2e.yml

name: e2e
on:
  pull_request:
permissions: {}
jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          name: e2e-evidence
          path: |
            playwright-report/
            test-results/
          retention-days: 30

The job needs no secrets: it runs against the app it started, with seeded data. Mark e2e as a required status check. To keep the required gate fast as the suite grows, run npx playwright test --grep @critical as the required job and the full suite on a schedule.

Adopt the E2E policy

Commit the policy next to the PR template, and merge it through the platform code owner.

docs/policies/e2e.md

# E2E policy for user-visible changes

Owner: VP Engineering. Enforced by: tech leads. Reviewed: quarterly. Version: 1.

## Scope
Every pull request that changes what a user sees or does, whoever wrote it.

## Evidence by risk class
Low: screenshots at 1280x720 and 390x844, no new console errors.
Medium: browser run per acceptance criterion with trace, plus a
component or integration test.
High: as medium, plus E2E happy path and main failure path in CI.
Critical (auth, payment, destructive, privacy): E2E with negative cases,
written or reviewed by a person; named approver before merge.

## Environment
Seeded test accounts and synthetic data only. The browser agent never
holds production credentials or writes to production.

## Retention
Evidence table in the PR; screenshots, traces and logs as CI artifacts
for 30 days. No secrets or customer data in any artifact.

## Flaky tests
A flaky test fails CI. Quarantine only with an owner, an issue and a
fix-by date; quarantined @critical tests block release until fixed.

## Protected tests
@critical tests change only in a PR whose purpose is that change,
approved by the flow's code owner. Agents do not edit them while
implementing a feature.

## Review
Quarterly: escaped UI defects, flaky and quarantined tests, share of
user-visible PRs with complete evidence, and critical flows without a test.

Copy-paste prompts for browser evidence

Paste these into Claude Code, Codex, or Cursor with the Playwright server configured. The first runs before implementation, the second after it, and the third when a flow is ready to become a test.

Copy-paste prompt: turn acceptance criteria into a browser checklist

Read the acceptance criteria in the spec for this change. For each criterion, write a browser check with: starting state, test account role, viewport (1280x720 and 390x844), the exact actions, the expected visible result, the expected backend effect you can observe (a network request, a stored record, an email in the test inbox), and one outcome that must not happen. Add at least one negative case per criterion. Assign the change a risk class (low, medium, high, critical) using docs/policies/e2e.md and list the evidence that class requires. Do not start implementing.

Copy-paste prompt: run the changed flow and retain evidence

Start the app with npm run start:test and seed the test accounts. Using the Playwright MCP server, run only the browser checks for this change, at both viewports. For every check, save a screenshot to .evidence/browser with the criterion ID in the file name, and record console errors and failed network requests. Use synthetic data only and never connect to a production URL. Finish with a markdown table for the PR: criterion, pass or fail, evidence file, console errors, failed requests. Mark anything you judged visually as "needs human review" rather than pass.

Copy-paste prompt: promote a stable flow to a deterministic test

From the passing browser checks for this change, pick the flows that guard revenue, access, or stored data. For each, write the smallest Playwright test in e2e/ that asserts the visible result and the backend effect, using getByRole or getByLabel locators and web-first assertions, with no fixed waits. Tag revenue and access flows @critical. Run npx playwright test --repeat-each 5 on the new tests and report any flaky result instead of adding retries. List what still needs human visual judgment. Do not modify any existing @critical test.

--repeat-each is a npx playwright test option; five green repeats locally is a cheap flake screen before the test becomes a required gate.

How do you know the policy works without watching every run?

You test the policy itself, then watch four numbers. Run these checks when you introduce the policy and after any change to the CI job or the Playwright configuration.

Self-test	How to run it	Expected result
Flake detection	Add a test that fails on its first attempt only	The job fails and the report marks the test flaky
Stray test.only	Commit a test.only on a branch	CI fails before running the suite
Wrong server	Start another app on the test port, then run CI locally with CI=1	Playwright refuses the port instead of testing the other app
Protected tests	Ask an agent to implement a feature that conflicts with an @critical assertion	The agent reports the conflict; any edit to the test needs the e2e/ code owner’s approval before merge (CODEOWNERS + required code-owner review)
Evidence completeness	Open a user-visible PR without the evidence table	The evidence bundle check fails
No production reach	Grep the agent environment and MCP configuration for production hosts and keys	Nothing found

Review four numbers each quarter: UI defects that escaped to production, the count and age of quarantined tests, the share of user-visible PRs with complete evidence, and the critical flows that still have no test. Define them once on the AI metrics panel. Rising escapes with complete evidence mean the criteria are too weak; rising quarantine age means flakes are being parked, not fixed.

What breaks in an E2E policy for agent-built UI?

Screenshots without a contract. A screenshot looks right while the action failed, the wrong account was used, or the backend never changed. Recovery: require every criterion to name a backend effect, and reject evidence that is only a picture.

Retries turn flakes into green. A suite with retries: 2 and no flake gate passes tests that fail a third of the time, and agents learn that red is transient. Recovery: set failOnFlakyTests, quarantine with an owner and a date, and treat a quarantined @critical test as a release blocker.

The agent edits the test to match the bug. Asked to “make E2E pass”, an agent relaxes an assertion or updates a visual baseline. Recovery: CODEOWNERS on e2e/ and baselines, the protected-test rule in the policy, and a revert of any baseline change that arrived inside a feature PR.

The browser agent reaches real data. A saved profile or a copied .env gives the agent a logged-in production session. Recovery: --isolated on every server, seeded accounts only, and no production credentials in the agent’s environment. If it happened, rotate the credentials and handle it through the agent incident runbook.

Parallel agents share one browser profile. Two worktrees using a persistent profile collide, and runs fail in ways that look like product bugs. Recovery: --isolated or a distinct --user-data-dir per worktree, and a distinct app port per worktree.

Visual judgment is treated as a pass. An agent that says “the layout looks correct” has given an opinion. Recovery: route layout and brand quality to a human reviewer with the screenshots, or to a calibrated rubric judge as on model-graded checks, and keep accessibility on automated semantic checks from accessibility testing.

Where to go next with E2E evidence

Place this rule in the test stage, then carry its evidence into review.

Test stage The evidence matrix for every kind of change.

Agent-driven browser verification The developer's loop behind this policy.

Evidence bundle Where the browser evidence lands in the PR.

Q9 · Review automation Carry browser evidence into the merge decision.

Q13 · AI in CI/CD Run agents inside the pipeline without handing them the gate.

CTO Scorecard answer key Every scorecard question and its canonical page.