Pipeline automation with AI

Pipeline automation with AI means a coding agent such as Claude Code, Codex or Cursor edits the CI configuration itself: change-aware jobs that skip untouched services, triage that separates flaky tests from real regressions, and repairs to broken workflows. The pipeline stays trustworthy only when linters, an always-run aggregate check and code owners verify every agent edit.

Your monorepo runs every job on every push. A one-line README change waits behind a full build of six services, the same three end-to-end specs fail on some runs and pass on a rerun, and everyone has learned to click Re-run failed jobs without reading the log. You want an agent to fix that, but CI is the gate that protects main, and an agent that quietly weakens the gate turns every later green check into a false one.

This page is for developers who own a repository’s CI and tech leads who decide which checks are required. It covers the agent editing the pipeline. Running an agent as a step inside the pipeline is on headless agents in CI, and rolling a merged change out to production is on progressive delivery for agent-written changes.

What you’ll walk away with from pipeline automation

• A change-aware ci.yml built on dorny/paths-filter, with least-privilege permissions, hash-pinned actions and one aggregate check to mark as required.
• A flaky-test triage loop that feeds a failed run’s log to the agent and confirms every “flaky” verdict by rerunning the spec before anything is quarantined.
• A workflow lint job (actionlint and zizmor) that checks every agent edit to .github/ without a human reading the YAML line by line.
• Three copy-paste prompts, per-tool commands for Claude Code, Codex and Cursor, and the failure modes that turn a green pipeline into a false signal.

What changes when an agent edits CI?

An agent writes plausible YAML quickly. The risk is not a syntax error, which CI reports at once. The risk is a pipeline that runs, goes green, and no longer checks what it used to check: a filter that skips the test job, a retry that hides a regression, or a permission that widens a token. DORA’s guidance on AI adoption puts version control in the same role: “As AI increases the volume and velocity of code generation, version control becomes your critical safety net” (Google Cloud blog, Nathen Harvey and Allison Park, 2025-12-10). CI is the part of that net that runs on every change.

So every workflow on this page follows the same order:

1. State the contract first. Tell the agent which checks must never be skipped (type check, lint, tests for any changed package) and which job is the required status check.
2. Let the agent edit. Use the prompt for the task, against your real directory layout.
3. Verify mechanically. The workflow lint job, the aggregate check and one deliberate test push prove the edit. You read the verification output, not every line of YAML.
4. Sign off by ownership. A CODEOWNERS entry on /.github/ routes every pipeline change to the person accountable for the gate.

Build only what changed with dorny/paths-filter

In a monorepo, the largest CI saving usually comes from not running jobs whose inputs did not change. dorny/paths-filter compares the diff with path patterns and exposes one output per filter, which later jobs read in their if: conditions. On pull_request it lists changed files through the GitHub API, so the job needs pull-requests: read. On push it uses git, so it needs a checkout.

The agent should produce a file shaped like this one, derived from your real workspaces rather than copied:

.github/workflows/ci.yml

name: CI
on:
  pull_request:
  push:
    branches: [main]

permissions: {}

jobs:
  changes:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: read
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
      workflows: ${{ steps.filter.outputs.workflows }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
        id: filter
        with:
          filters: |
            api:
              - 'services/api/**'
              - 'packages/shared/**'
              - 'package-lock.json'
              - '.github/workflows/**'
            web:
              - 'services/web/**'
              - 'packages/shared/**'
              - 'package-lock.json'
              - '.github/workflows/**'
            workflows:
              - '.github/**'

  test-api:
    needs: changes
    if: ${{ needs.changes.outputs.api == 'true' }}
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - run: npm ci && npm test --workspace=services/api

  # test-web: the same shape, gated on needs.changes.outputs.web

  required:
    if: ${{ always() }}
    needs: [changes, test-api]
    runs-on: ubuntu-latest
    steps:
      - name: Fail if any needed job failed or was cancelled
        if: ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}
        run: exit 1

Four details in this file are the ones agents most often get wrong, so name them in your prompt:

• Shared inputs appear in every filter. packages/shared/**, the lockfile and the workflow directory rebuild every consumer. A change to CI itself therefore runs the whole suite once.
• The comparison is explicit. Filter outputs are the strings 'true' and 'false', so the condition compares with == 'true'.
• One aggregate job is the required check. GitHub’s documentation states that a skipped job reports its status as “Success” and does not block a merge, even when it is a required check. Mark only required as required in branch protection. It runs on every event, and it fails when any job it needs failed or was cancelled.
• Actions are pinned to a commit. The comment keeps the version readable. zizmor, used in the lint job below, requires hash pins on all actions by default since v1.20.0.

The interaction differs by tool. Cursor edits the file in the editor and shows a diff. Claude Code and Codex can run headless from the terminal and lint the result in the same session.

Claude Code

Run it in print mode from the repository root, with actionlint installed locally (brew install actionlint). acceptEdits lets the agent write files without prompting, and the allowlist lets it run actionlint on its own output:

claude -p "$(cat prompts/change-aware-ci.md)" \
  --permission-mode acceptEdits \
  --allowedTools "Bash(actionlint *)"

Save the copy-paste prompt below as prompts/change-aware-ci.md so the team reruns the same instruction when a service is added. To reuse it inside interactive sessions, turn it into a project skill under .claude/skills/.

Codex

Run it non-interactively with the workspace permission profile, which lets Codex edit files in the repository (permission profiles are beta in Codex CLI 0.157.1):

codex exec -c default_permissions=":workspace" \
  -o ci-change-summary.md \
  "$(cat prompts/change-aware-ci.md)"

-o writes the agent’s final message to a file, which you can paste into the pull request description. To run the same task in Codex cloud against a branch, use codex cloud exec --env ENV_ID --branch BRANCH "…", where ENV_ID is an environment listed by codex cloud.

Cursor

Open the repository, start a new agent chat and paste the prompt below. Cursor reads the workspaces field and the services/ tree, then proposes .github/workflows/ci.yml as a diff. Before you accept, check the four details listed above in the diff view. They are the whole review surface for this change. Use Plan Mode first if the repository has more than one existing workflow, so the agent lists the jobs it intends to gate before it edits.

Copy-paste prompt: change-aware monorepo CI

Read the "workspaces" field in package.json and every existing file in .github/workflows/.
Rewrite .github/workflows/ci.yml so that:
1. A "changes" job uses dorny/paths-filter pinned to commit ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d (# v4.0.3) and actions/checkout pinned to 3d3c42e5aac5ba805825da76410c181273ba90b1 (# v7.0.1) with persist-credentials: false.
2. There is one filter per workspace. Every filter also matches packages/shared/**, the lockfile and .github/workflows/**. Add a "workflows" filter matching .github/**.
3. Each existing build, lint and test job gets "needs: changes" and "if: ${{ needs.changes.outputs.<name> == 'true' }}". Always compare with == 'true'.
4. The workflow has "permissions: {}" at the top and each job declares only what it needs (changes: contents read, pull-requests read).
5. A final job named "required" runs with if: ${{ always() }}, needs every other job, and exits 1 if any result is failure or cancelled.
Do not change the steps inside existing jobs. Run actionlint on the result and fix every finding.
Finish with a table: for each of these changed paths, which jobs run?
README.md, services/api/src/index.ts, packages/shared/util.ts, package-lock.json, .github/workflows/ci.yml

Verify the result without reading every line. The table at the end of the prompt is the agent’s claim about behaviour. Test that claim on the pull request: push one commit that touches only README.md and check the job states with gh pr checks. test-api should show as skipped, required should pass, and a commit to packages/shared/ should run every test job. If any row in the table disagrees with what CI did, the filter is wrong, whatever the YAML looks like.

Triage flaky tests without hiding regressions

A test that fails on some runs and passes on others is either flaky or a real intermittent bug, such as a race condition in the product code. Blanket retries treat both the same way, and that hides the second kind. The agent’s job is to classify each failure, and your job is to confirm each “flaky” verdict with a rerun before anything is quarantined.

Get the failed log into the agent first. The GitHub CLI prints only the failed steps of a run:

RUN_ID=$(gh run list --workflow ci.yml --status failure --limit 1 \
  --json databaseId --jq '.[0].databaseId')
gh run view "$RUN_ID" --log-failed > failed.log

With the GitHub MCP server connected, the agent can fetch the same log itself when you name the run. That saves the copy step in an interactive session. In a script, the gh pipe is simpler and needs no token in the agent’s configuration.

Claude Code

Pipe the log in and restrict the agent to reading files and running the test runner, so the classification cannot turn into edits. --permission-mode dontAsk keeps that true on every provider, because -p starts in auto mode on third-party providers or with telemetry off (from v2.1.285):

claude -p "$(cat prompts/flaky-triage.md)" \
  --permission-mode dontAsk \
  --allowedTools "Read,Grep,Bash(npx playwright test *)" < failed.log

Codex

codex exec appends piped standard input to the prompt as a <stdin> block. The read-only profile lets it inspect and run commands without writing to the repository:

codex exec -c default_permissions=":read-only" \
  "$(cat prompts/flaky-triage.md)" < failed.log

Rerunning Playwright writes test results to disk, which the read-only profile blocks. If the confirmation reruns fail on write errors, switch to :workspace for this step only.

Cursor

Attach failed.log to a new agent chat, or ask the agent to read the run through the GitHub MCP server, then paste the prompt below. Cursor runs the confirmation commands in its terminal; whether it asks before each one depends on your run mode settings.

Copy-paste prompt: classify, confirm, then quarantine

The attached log is from our last failed CI run.
Step 1, classify: for each failing test, decide "flaky" (timeout, test-order dependence, shared state between tests, clock or network dependence in the test itself) or "real" (the assertion reflects wrong product behaviour, including an intermittent race in product code). Quote the log line that supports each verdict.
Step 2, confirm: for every test you called flaky, run it 20 times in isolation, for example "npx playwright test <file> -g '<title>' --repeat-each=20". Report passes and failures per test. If a "flaky" test fails 3 or more times in 20, reclassify it as real.
Step 3, report only. Do not edit any file. Output a table: test, verdict, evidence, suggested fix. For each confirmed flaky test, propose a quarantine: a skip marker with a comment naming the run ID and today's date, and a retry setting scoped to that one test file, never to the global config.

Save this prompt as prompts/flaky-triage.md.

Apply the quarantine as a separate change, reviewed like any other change to the test oracle. Keep the quarantined tests in a list with a date and an owner. A quarantine with no owner never ends.

Lint every agent edit to the pipeline

Workflow files have two classes of defect that tests cannot catch: expressions that do not evaluate the way the author thinks, and security problems such as template injection, over-broad tokens or unpinned actions. Two static analysers cover them. actionlint type-checks workflow syntax and expressions. zizmor audits workflows for security issues. Both run in seconds, so run them on every change to .github/, gated by the workflows filter from the first section. Add this job to ci.yml and add it to the needs list of required:

  lint-workflows:
    needs: changes
    if: ${{ needs.changes.outputs.workflows == 'true' }}
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: actionlint
        run: |
          bash <(curl -sSfL https://raw.githubusercontent.com/rhysd/actionlint/v1.7.12/scripts/download-actionlint.bash) 1.7.12
          ./actionlint -color
      - name: zizmor
        run: pipx run --spec zizmor==1.30.1 zizmor --offline --format=github .github/workflows/

This job runs on pull_request with a read-only token and executes nothing from the contributor’s branch; it only runs two pinned linters over the workflow files, so it is safe for pull requests from forks. Keep it that way: an agent asked to “make the checks faster” sometimes switches a workflow to pull_request_target, which runs with the base repository’s secrets. The rules for running an agent next to secrets are on headless agents in CI.

One line breaks the job’s own hash-pinning rule on purpose: the actionlint download script is fetched by its release tag, and the binary it installs is fixed by the version argument 1.7.12. Treat it as an accepted exception, or vendor the script into your repository if your policy allows none.

Then route ownership. One line in .github/CODEOWNERS, plus Require review from Code Owners in branch protection, sends every pipeline change to the people accountable for the gate:

/.github/ @your-org/platform

The code owner does not read the YAML line by line. They read three things: the lint job’s result, the agent’s “which jobs run” table next to what gh pr checks showed, and the diff of the required job’s needs list, which must never shrink without a reason in the pull request description.

Copy-paste prompt: review an agent's pipeline change

Compare .github/ on this branch with origin/main. Report, as a checklist:
1. Any job removed from the "needs" list of the "required" job, or any job whose "if:" condition got narrower.
2. Any permissions added or widened, any new secret reference, any trigger changed to pull_request_target, workflow_run or issue_comment.
3. Any action reference that is not pinned to a full commit SHA.
4. Any change to test selection: skip markers, retries, testPathIgnorePatterns, grep or exclude options, or deleted test files.
5. The output of "actionlint" and "zizmor --offline .github/workflows/".
For each finding, quote the line and say whether it weakens the gate. Do not edit files.

Paste this prompt into an interactive session on the pull request branch, or save it as prompts/pipeline-review.md so every reviewer runs the same checklist.

Where does the deploy step fit?

Keep deployment out of the agent’s hands and out of pull requests. Run the deploy job only on push to the protected default branch, behind a GitHub environment with its own secrets and required reviewers. An agent can write and change that job like any other workflow file, under the same lint job and code owners, but no agent session should hold the deploy credentials.

What happens after the deploy job starts is progressive delivery: canaries, feature flags and automatic rollback on a service-level signal. That page covers the rollout controllers and the blast-radius budget per risk class. For the tool-specific deploy jobs, see pipeline configuration with Claude Code, Codex in CI/CD with the GitHub Action and CI/CD pipelines with Cursor.

Which model should edit CI?

Pipeline edits need cross-file reasoning, because a filter is only correct relative to the package graph. Start on the tool’s default model and raise effort before you switch. As of 2026-10-12 that means Claude Opus 5.5 in Claude Code from v2.1.280 (on both release channels; raise effort with /effort) and GPT-6.1 Sol in Codex (bundled default since CLI 0.159.1). Cursor names no single default model (its models and pricing page, checked 2026-10-12), so check the model picker in your own Cursor install. Prices and the full lineup are on the models hub.

What breaks when an agent edits CI?

The filter condition is always true

if: ${{ needs.changes.outputs.api }} without a comparison is truthy for the non-empty string 'false', so the job runs on every change and the saving disappears. Recovery: require == 'true' in every condition. The docs-only test push shows the problem at once, because the job runs when it should be skipped.

A skipped job merges a broken change

A filter that misses a dependency skips the test job, and a skipped required check counts as a success. The pull request merges green with no tests run. Recovery: make required the only required check, keep shared paths, the lockfile and .github/workflows/** in every filter, and add a row to the agent’s “which jobs run” table for each new shared package.

The agent turns red CI green by weakening the test

Asked to “fix the failing build”, an agent can add a skip marker, raise a timeout, widen a retry or delete an assertion. CI goes green and the regression ships. Recovery: treat any change to test selection or assertions as an oracle change that needs a named human approver, and check for it with the review prompt above. Reviewing an agent’s pull request covers oracle changes in detail.

Global retries hide a real race

retries: 2 in the global test config lets an intermittent product bug pass on the second attempt. Recovery: scope retries to the quarantined files only, confirm each “flaky” verdict with repeated runs, and give every quarantine an owner and a date.

The fix-the-YAML loop

Asking the agent to “fix the YAML” without the error sends it into rewrites of lines that were already correct. Recovery: paste the exact actionlint output or the Actions annotation, and tell the agent to change only the lines it names.

Where to go next with pipeline automation

The evidence bundle Make CI fail any agent pull request that does not prove what it claims.

Headless agents in CI Run claude -p and codex exec as pipeline steps, with the secret-handling rules.

Progressive delivery Canaries, flags and automatic rollback after the deploy job starts.

CI/CD pipeline patterns Recipes for generating whole pipelines, proving their gates block, and GitLab CI or Jenkins.