Skip to content
AI Developer Toolkit

Find your level — free Buy the Toolkit Buy Toolkit Buy the Toolkit Buy Toolkit Open the guides Read
Navigation menu
Buy the Toolkit Find your level — free Buy the Toolkit Open the guides Sign in

Test data strategies and fixtures

PL

Level 3 How-to Updated

Test data management for agent-written tests means replacing shared, hand-edited fixtures with three generated assets: factories built from the schema, named scenario seeds for integration and end-to-end runs, and PII-free synthetic datasets. An agent can write all three, but they are trustworthy only when contract tests, a seed-diff check and a PII gate prove them in CI.

This page is for developers who own a test suite and tech leads who own its seeds. The situation is familiar: a seed file copied from production years ago, 40 tests that assume user ID 1 is “John Doe”, and an agent that “fixes” a failing test by adding the row it needs to the seed. Every new unique constraint breaks a batch of tests, and nobody can say which rows are still load-bearing.

What you get from agent-built test data

Section titled “What you get from agent-built test data”
  • A rules block for AGENTS.md or CLAUDE.md that makes every factory deterministic, collision-free and PII-free.
  • A prompt that generates schema-derived factories with traits, and the command to run it headless in Claude Code, Codex or Cursor.
  • A scenario-seed design that maps named rows to acceptance criteria, so end-to-end tests stop depending on “whatever is in the database”.
  • A prompt for production-shaped synthetic data that never shows the agent a real row.
  • Four CI checks that prove the data layer without anyone reading the factories: a factory contract test, a seed-determinism diff, a PII gate and CODEOWNERS on the data paths.

Which test data does each test layer need?

Section titled “Which test data does each test layer need?”

Decide the data source per layer before you prompt. An agent left to choose picks one pattern and applies it everywhere, usually database inserts in unit tests.

LayerData sourceIsolationOwner
Unitbuild(): plain objects from a factory, no databaseNone needed; nothing persistsWhoever writes the test
Integrationcreate(): factory rows inside a transaction or per-worker schemaRollback per test, or a schema per worker for code that commitsTest author, on the harness from integration testing
End to endA named scenario seed loaded once per run, plus rows tagged per testA fresh database per run or per agent taskTech lead (CODEOWNERS)
Staging and demosA generated synthetic dataset with production-shaped distributionsA separate database, rebuilt by a jobTech lead or platform team

The ownership column matters more with agents than with people. A test that fails because a row is missing gives the agent an easy way out: add the row. Unit and integration data are created by the test itself, so that shortcut is visible in the diff. Shared seeds are not, which is why they get an owner.

Set the test data rules before the agent writes a factory

Section titled “Set the test data rules before the agent writes a factory”

Agents reproduce the patterns in their training data, and several of those patterns cause flaky tests. Put the rules where every session reads them. The following block is ready to paste into AGENTS.md (read by Codex, and by Claude Code 2.1.277+ when there is no CLAUDE.md) or CLAUDE.md; in Cursor, add the same text as a project rule:

## Test data rules
- Test data comes from factories in tests/factories/. Never hardcode IDs, and never
  depend on rows another test created.
- Seed Faker once per test file from TEST_SEED (default 20260926) and print the seed
  when a test fails. Freeze Faker's reference date with setDefaultRefDate.
- Unique columns use a per-worker sequence plus a per-file run ID
  (`user-${workerId}-${runId}-${seq}@example.com`), not Faker values. Vitest
  reloads factories in every test file, so the counter alone repeats across files.
  faker.helpers.unique does not exist in @faker-js/faker 10.
- Emails use example.com, example.net or example.org only. No real names, addresses
  or phone numbers copied from anywhere.
- Traits encode business rules: a 'delivered' order has shippedAt < deliveredAt; an
  'outOfStock' product has stock = 0. Totals are computed from line items.
- Do not edit tests/factories/, tests/seeds/ or the seed scripts to make a failing
  test pass. Report the failure instead.

Two of these rules exist because of specific failure patterns. Random values for unique columns collide once a suite grows or runs on parallel workers, and the resulting failure appears on an occasional CI run, which is the hardest kind to diagnose. faker.helpers.unique was the old answer, but it is absent from @faker-js/faker 10.6.0 (checked 2026-09-26), and agents still write it from older examples. A sequence that includes the worker ID and a per-file run ID stays unique even when rows are committed, which matters on the schema-per-worker path: Vitest reloads the factory module, and with it the counter, in every test file. Every other value still comes from the seeded Faker, so a failing seed reproduces.

Generate factories from the schema

Section titled “Generate factories from the schema”
  1. Save the factory prompt below as prompts/factories.md, and edit the entity list and the traits for your domain.
  2. Run it in your tool (tabs below). The agent reads the schema, writes one factory per entity and a contract test for each.
  3. Run the contract tests yourself and read the report, not the factories. The verification section below explains what each check proves.
  4. Commit the factories and the contract tests in one pull request. After that, ask for new tests “using the factories in tests/factories/”.

Run it interactively with claude, or headless when you want the same result every time. --allowedTools pre-approves only what the task needs (flags checked with claude --help on 2.1.283, 2026-09-26):

Terminal window
# terminal, repository root
claude -p "$(cat prompts/factories.md)" \
  --allowedTools "Bash(npx vitest run *)" Edit Write

A generated factory should look roughly like this. The shape is what you check, not every line:

tests/factories/user.factory.ts
import { faker } from '@faker-js/faker';
import { users, type NewUser } from '../../src/db/schema';
import { randomUUID } from 'node:crypto';
import type { Tx } from '../integration/db';


// Vitest loads this module once per test file, so seq restarts at 0 in every file.
// runId keeps committed rows from two files on the same worker apart.
const workerId = process.env.VITEST_POOL_ID ?? '0';
const runId = randomUUID().slice(0, 8);
let seq = 0;


export const userTraits = {
  admin: { role: 'admin' },
  suspended: { status: 'suspended', suspendedAt: faker.date.recent({ days: 30 }) },
  newToday: { createdAt: faker.defaultRefDate() },
} satisfies Record<string, Partial<NewUser>>;


export function buildUser(overrides: Partial<NewUser> = {}, trait?: keyof typeof userTraits): NewUser {
  seq += 1;
  return {
    email: `user-${workerId}-${runId}-${seq}@example.com`,
    name: faker.person.fullName(),
    role: 'member',
    status: 'active',
    createdAt: faker.date.past({ years: 2 }),
    ...(trait ? userTraits[trait] : {}),
    ...overrides,
  };
}


export async function createUser(tx: Tx, overrides: Partial<NewUser> = {}, trait?: keyof typeof userTraits) {
  const [row] = await tx.insert(users).values(buildUser(overrides, trait)).returning();
  return row;
}

Seed Faker in the test setup file so a failure can be replayed with the same data:

// tests/setup.ts (listed under setupFiles in the Vitest config)
import { faker } from '@faker-js/faker';


const seed = Number(process.env.TEST_SEED ?? 20260926);
faker.seed(seed);
faker.setDefaultRefDate(new Date('2026-01-01T00:00:00Z'));
console.info(`faker seed ${seed}; replay with TEST_SEED=${seed}`);

For Python projects the same design maps onto factory_boy (3.3.3 on PyPI) with Faker (40.40.0); for TypeScript, fishery (2.4.0 on npm) is an option if you prefer a library to hand-written builders. Versions checked 2026-09-26.

Give the agent the live schema, not only the migration files

Section titled “Give the agent the live schema, not only the migration files”

A factory that matches schema.ts can still fail on the real database: a CHECK constraint added in a raw SQL migration, a trigger, or a column default the ORM does not know about. A database MCP server lets the agent query information_schema on the test container and try its inserts before it reports. The integration testing page shows how to connect DBHub to a throwaway test container in all three tools; database MCP servers compares the alternatives and their read-only switches. Do not use @modelcontextprotocol/server-postgres: npm marks it “no longer supported” and it sits in the archived reference servers (checked 2026-09-26). Never point any of them at production for this work.

Design scenario seeds for integration and end-to-end tests

Section titled “Design scenario seeds for integration and end-to-end tests”

A scenario seed is a small, named dataset in which every row exists because an acceptance criterion needs it. “An expired subscription that tries to export” is a scenario. “Ten random users” is not, because no test can say which of them it depends on.

Write the seed as code that uses the factories, and export a handle for every named row. Tests then refer to scenario.users.expired, never to an ID or an email. Tag rows a test creates on top of the seed with the test’s name, so cleanup can delete them without touching the reference data.

If seeding takes more than a few seconds, seed a template database once and create each run’s database from it with Postgres’s CREATE DATABASE run_db TEMPLATE app_template. The ephemeral environments page wires that into a per-task stack.

Known credentials for end-to-end logins belong in the seed too, but generate them per run and read them from the seed’s handles, not from a README. A fixed password such as TestAdmin123! ends up in screenshots, traces and eventually a shared staging environment.

Generate production-shaped synthetic data without PII

Section titled “Generate production-shaped synthetic data without PII”

Staging, demos and performance tests need data that behaves like production: realistic tier mixes, long-tail order values and the odd pathological record. The safe way to get it is to give the agent the shape of production, never its rows. A human, or a reviewed job, runs aggregate queries (counts per tier, order-value percentiles, share of refunds) and hands the agent those numbers. The agent writes a seeded generator that reproduces them with synthetic values. The organizational policy behind this rule is on data privacy and enterprise policies for coding agents.

The summary table is the verification: you compare target and generated aggregates, not rows. For reproducing one production bug, generate the specific record shape (the unicode name, the null field, the boundary date) from the bug report instead of anonymizing the customer’s real record. An anonymized record still carries structure that can identify someone.

How do you know the test data is right without reading every factory?

Section titled “How do you know the test data is right without reading every factory?”

You do not review 30 generated factories line by line. You review four checks that fail when the data layer is wrong, and CI runs them on every pull request.

1. Factory contract test. The test the factory prompt asked for. Inserting every trait into the real schema makes the database the validator for types, nullability, foreign keys and CHECK constraints. The uniqueness assertion goes red the day someone swaps the sequence for a random email:

// tests/factories/factories.contract.test.ts (excerpt)
import { describe, it, expect } from 'vitest';
import { buildUser, createUser, userTraits } from './user.factory';
import { withRollback } from '../integration/db';


describe('user factory', () => {
  it('builds 1,000 users with unique example.com emails', () => {
    const emails = Array.from({ length: 1000 }, () => buildUser().email);
    expect(new Set(emails).size).toBe(1000);
    expect(emails.every((e) => /@example\.(com|net|org)$/.test(e))).toBe(true);
  });


  it.each(Object.keys(userTraits))('inserts trait %s into the real schema', (trait) =>
    withRollback(async (tx) => {
      const row = await createUser(tx, {}, trait as keyof typeof userTraits);
      if (trait === 'suspended') expect(row.suspendedAt).not.toBeNull();
    }));
});

Trait rules that span many values, such as “timestamps always increase” or “total equals the sum of line items”, are better checked as properties over generated inputs; see property-based testing.

2. Seed-determinism diff. Seed twice from scratch and compare the dumps. Any difference is non-determinism that will return as a flaky test. db:reset and db:seed stand for your project’s own scripts. Since the August 2025 minor releases (17.6, 16.10, 15.14, 14.19, 13.22), pg_dump writes a randomly keyed \restrict line into every plain-format dump, so two dumps of identical data always differ unless you pass a fixed --restrict-key (older pg_dump builds reject the flag; drop it there):

Terminal window
# CI or terminal, against the throwaway test database only
npm run db:reset && npm run db:seed
pg_dump --data-only --inserts --restrict-key=seedcheck "$TEST_DATABASE_URL" > /tmp/seed-a.sql
npm run db:reset && npm run db:seed
pg_dump --data-only --inserts --restrict-key=seedcheck "$TEST_DATABASE_URL" > /tmp/seed-b.sql
diff /tmp/seed-a.sql /tmp/seed-b.sql && echo "seed is deterministic"

3. PII gate. A cheap check that fails the build when a non-example email appears in test data. It catches the most common leak, an address pasted from a support ticket. It does not detect names or addresses, which is why the synthetic-data prompt never receives rows in the first place:

Terminal window
# CI step: fails if any email outside example.com/.net/.org is found
! grep -rEoh '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[a-z]{2,}' tests/ seed-output/ \
  | grep -vE '@example\.(com|net|org)$'

4. Ownership. Put tests/factories/, tests/seeds/ and the seed scripts under CODEOWNERS with the tech lead as owner, and keep the “do not edit to make a test pass” rule in AGENTS.md. A pull request in which the agent changed a seed and a failing test together is the pattern to reject. Protect the oracle covers hooks that block those edits during the session.

Sign-off follows from that split. CI signs off the contract test, the seed diff and the PII gate. The tech lead signs off changes to factories and seeds through CODEOWNERS. The test author signs off test-local data. To judge whether the suite built on this data actually catches bugs, use the measures in oracle strength.

What breaks when agents manage test data?

Section titled “What breaks when agents manage test data?”

The agent calls faker.helpers.unique and the suite fails to compile. The helper is gone in @faker-js/faker 10. Recovery: point the agent at the rules block and ask it to replace every call with the per-worker sequence, then run the contract test.

A test fails on an occasional CI run and never locally. Usually an unseeded Faker call, new Date() inside a factory, or a unique value that collides across parallel workers. Recovery: rerun with the TEST_SEED printed in the failing log. If it reproduces, fix the factory; if not, look for a time or worker dependency and move it behind setDefaultRefDate or the worker ID.

The agent adds a row to the shared seed to make its test pass. The test now passes for a reason unrelated to the code. Recovery: revert the seed change, ask for the row to be created inside the test through a factory, and add CODEOWNERS to the seed paths if the edit got through review.

Factories drift from the schema after a migration. New NOT NULL columns break every create(). Recovery: the contract test fails first and names the column. Rerun the factory prompt with “update the existing factories for the migration in migrations/<file>; add defaults for new required columns”.

Deep foreign-key chains make integration tests slow. Each createOrder() creates a user, an organization and a plan. Recovery: create shared reference data (plans, roles, countries) once per worker in beforeAll, pass parents in explicitly where a test creates many children, and batch inserts with createXList.

Someone pastes a customer record into the prompt to reproduce a bug. That data is now in the session transcript and possibly the vendor’s logs. Recovery: log it as a data incident and request deletion as described under data privacy and enterprise policies for coding agents, then regenerate the record’s shape synthetically with the approach above.

Where to go next with test data

Section titled “Where to go next with test data”
Integration testing Prerequisite: the Testcontainers harness and database MCP setup these factories run on.
Ephemeral environments Load scenario seeds into a per-task stack with template databases.
Property-based testing Check trait rules and invariants over generated inputs.
Oracle strength Measure whether the suite built on this data catches bugs.
Data privacy for coding agents The policy for what data may reach an agent at all.
All 577 guides Free resources: Compare tools Glossary By role

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings llms.txt pricing.md RSS