Skip to content
AI Developer Toolkit

AI in CI/CD: a bounded server-side loop

AI in CI/CD is safe when a coding agent runs as a bounded server-side job: a trusted trigger sets the scope, the agent works without write credentials and hands over a patch, deterministic checks decide what is green, and a named human keeps merge and production authority. An agent that can push, merge, or edit its gates fails.

This page is for the CTO who owns question 13 of the CTO Scorecard and for the platform lead who builds the pipeline. The usual situation: someone added an AI job to GitHub Actions last quarter. It started as PR summaries, then gained Bash and write access “to fix lint”, and now nobody can say which token it holds, what text it reads, or what stops it from editing the workflow that tests it.

Q13 · Quality gates: Are AI agents part of CI/CD itself, beyond developers running them locally?

Max-score answer: an artifact-driven pipeline in which the agent prepares, reviews, and tests; deterministic checks gate the pull request; and a named human retains the required merge and production authority.

What this CI/CD agent policy gives you

Section titled “What this CI/CD agent policy gives you”
  • A task table that decides which CI jobs an agent may run, on which trigger, and with which output
  • A reference workflow that splits the agent job (no write token) from the job that opens a draft pull request, with Claude Code and Codex variants
  • A policy file you can commit as-is and adapt in one review
  • Three copy-paste prompts: implement an accepted spec, triage a CI failure, and audit the final diff against the spec
  • A self-test that proves the boundaries hold without anyone reading the agent’s code
  • The failure modes seen in real incidents, each with a recovery step

The developer-level pipeline work (change-aware builds, flaky-test triage, deploy YAML) lives on the CI/CD pipeline patterns page. This page covers only the organizational boundary around an agent that runs inside CI. Read the agent threat model first if you have not classified what your agents read and can reach.

How do you score each level of Q13?

Section titled “How do you score each level of Q13?”

The scorecard awards 0–3 points. Each step up adds a control, not a feature.

PointsWhat the scorecard seesWhat moves you to the next level
0Agents run only on developer machinesPick one read-only CI task from the table below and run it on pull requests
1A single AI step, such as a PR summaryGive the agent a task that produces a change, with an explicit token scope and a job timeout
2An agent action auto-fixes lint or runs a security scanSeparate the agent from write credentials, route output through a draft PR, and protect the gates from agent edits
3Artifact-driven loop: trusted trigger, patch artifact, deterministic gates, named human for merge and productionKeep it there with the self-test and the monthly review below

Which CI tasks should an agent run?

Section titled “Which CI tasks should an agent run?”

Choose tasks by what the agent reads and what it can change. Untrusted text (issue bodies, PR comments, logs, fetched pages) and write access must never meet in one job. That combination is how a prompt in a GitHub issue title reached an issue-triage workflow running claude-code-action with Bash, Write and Edit, and ended in stolen publish tokens; this is the “Clinejection” chain, disclosed 2026-02-09 and reported by Adnan Khan and Simon Willison (secondary reports).

TaskTriggerAgent readsAgent may changeOutputWho decides
PR review commentspull_requestThe diff and the repositoryNothingReview commentsCode owner
CI failure triageFailed run on a trusted branchLogs and codeNothingA classified report and a proposed patch as an artifactTeam on call
Implement an accepted specworkflow_dispatch by a maintainerA merged spec on the default branchFiles in the checkoutPatch, then a draft PRCode owner merges
Dependency or maintenance classschedule (runs only from the default branch)Lockfiles, changelogsFiles in the checkoutPatch, then a draft PRCode owner merges
Anything touching .github/, deploy config, secrets, or test oraclesNot delegated——A human writes itPlatform code owner

Deployment is not in the table. Promotion and rollback stay with the release process on the Deploy stage page, and a production change keeps its own production approval gate.

How do you build the bounded pipeline?

Section titled “How do you build the bounded pipeline?”

  1. Accept a trusted trigger. Use workflow_dispatch (only users with write access can run it), a schedule, or a label that only maintainers can apply. The agent reads its instructions from files merged to the default branch, not from the text of the event.

  2. Run the agent without write credentials. The agent job gets permissions: contents: read, persist-credentials: false, a timeout-minutes value, a turn or budget cap, and a pinned CLI version. No production credentials exist in that job.

  3. Hand over an artifact, not a push. The agent leaves its changes in the workspace; the next step packages them as agent.patch with the run log and uploads both.

  4. Open the draft PR in a separate job with no model. That job applies the patch, refuses it if it touches CI, ownership, or agent configuration, and opens a draft pull request with a GitHub App token that has no workflows permission.

  5. Let deterministic checks decide. The normal CI runs on the draft PR: types, tests, lint, security scans, and the evidence bundle check. The agent may diagnose a red check in a new run; it may not redefine it as green.

  6. Keep human authority where it was. A named code owner marks the PR ready and merges. Nothing in this workflow can merge or deploy.

Run the agent job in each tool

Section titled “Run the agent job in each tool”

Steps 1 and 3–6 are identical for every tool. Only the agent job differs. Both examples implement a spec under changes/SLUG/spec.md that was merged to main through the plan policy, and both pin the model explicitly so that a vendor default change does not change CI behaviour; see the models hub before you change the ID.

Claude Code runs headless with claude -p. Every flag below is in claude --help (checked in 2.1.283 and 2.1.285) except --max-turns, which Anthropic’s GitHub Actions documentation lists as a supported CLI argument. In -p mode nobody can answer a permission prompt, so a call outside the allow rules is refused; --tools removes every other tool from the session. Hooks and settings in the checked-out repository’s .claude/ still load, which is one reason the draft-PR job refuses agent changes to that directory.

.github/workflows/agent-implement-spec.yml
name: agent-implement-spec
on:
  workflow_dispatch:
    inputs:
      spec:
        description: "Accepted spec on main, e.g. changes/rate-limit/spec.md"
        required: true
permissions: {}
concurrency:
  group: agent-${{ inputs.spec }}
  cancel-in-progress: false


jobs:
  agent:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - name: Accept only a merged spec under changes/
        env:
          SPEC: ${{ inputs.spec }}
        run: '[[ "$SPEC" =~ ^changes/[a-z0-9-]+/spec\.md$ ]] && test -f "$SPEC"'
      - uses: actions/setup-node@v7
        with:
          node-version: 24
      - run: npm ci
      - name: Run Claude Code headless
        env:
          ANTHROPIC_API_KEY: ${{ secrets.CI_AGENT_ANTHROPIC_KEY }}
          SPEC: ${{ inputs.spec }}
        run: |
          npm install -g @anthropic-ai/claude-code@2.1.283
          claude -p "$(cat .github/agent-prompts/implement-spec.md) Spec: $SPEC" \
            --model claude-opus-5-5 \
            --permission-mode acceptEdits \
            --tools "Read,Edit,Write,Glob,Grep,Bash" \
            --allowedTools "Read,Edit,Write,Glob,Grep,Bash(npm test *),Bash(npm run lint *)" \
            --max-turns 40 \
            --max-budget-usd 10 \
            --output-format json > "$RUNNER_TEMP/run.json"
      - name: Package the change
        run: |
          git add -A
          git diff --cached --binary > "$RUNNER_TEMP/agent.patch"
      - uses: actions/upload-artifact@v7
        with:
          name: agent-output
          path: |
            ${{ runner.temp }}/agent.patch
            ${{ runner.temp }}/run.*

The API key is in the environment of a process that runs npm test, so use a dedicated key with a spend limit for CI only. Bash(npm test *) runs scripts the agent can edit, so treat it as arbitrary execution. Keep repository secrets out of the CI jobs that run on agent/* branches, or add package manifests (package.json, lockfiles) to the protected paths. To avoid a static key, run the step through anthropics/claude-code-action@v1 with workload identity federation to the Claude API, or with use_bedrock / use_vertex / use_foundry and OIDC, as Anthropic’s GitHub Actions guide describes.

Open the draft pull request without the model

Section titled “Open the draft pull request without the model”

This job is the same for every tool. It has no model, no API key, and only the permissions it needs to open a pull request.

  open-draft-pr:
    needs: agent
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/download-artifact@v8
        with:
          name: agent-output
          path: ${{ runner.temp }}/agent
      - name: Apply the patch and refuse protected paths
        run: |
          git apply "$RUNNER_TEMP/agent/agent.patch"
          git add -A
          # add your deploy config, test-oracle paths and package manifests, e.g. |^deploy/|^tests/golden/|package\.json$|package-lock\.json$
          if git diff --cached --name-only | grep -E '^(\.github/|CODEOWNERS$|\.claude/|\.codex/|\.cursor/|AGENTS\.md$|CLAUDE\.md$)'; then
            echo "The agent changed CI, ownership, or agent configuration. A human must do this." >&2
            exit 1
          fi
      - uses: actions/create-github-app-token@v3
        id: app
        with:
          client-id: ${{ vars.CI_AGENT_APP_CLIENT_ID }}
          private-key: ${{ secrets.CI_AGENT_APP_KEY }}
          permission-contents: write
          permission-pull-requests: write
      - name: Push a branch and open a draft PR
        env:
          GH_TOKEN: ${{ steps.app.outputs.token }}
          SPEC: ${{ inputs.spec }}
        run: |
          branch="agent/${GITHUB_RUN_ID}"
          git switch -c "$branch"
          git -c user.name="ci-agent" -c user.email="ci-agent@users.noreply.github.com" \
            commit -m "agent: implement $SPEC"
          git push "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "$branch"
          gh pr create --draft --head "$branch" --title "agent: $SPEC" \
            --body "Spec: $SPEC. Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"

The GitHub App token matters twice. Its installation has no workflows permission, so even a patch that slipped past the path check could not push workflow changes. And a pull request opened with the default GITHUB_TOKEN does not start your CI workflows, so the checks in step 5 would never run.

Adopt the CI agent policy

Section titled “Adopt the CI agent policy”

Commit the policy beside the workflows it governs, and merge it through the platform code owner.

docs/policies/ci-agents.md
# CI agent policy


Owner: VP Engineering. Operated by: platform team. Reviewed: monthly. Version: 1.


## Allowed tasks
PR review comments, CI failure triage, implementing a merged spec under
changes/, and scheduled dependency maintenance. Anything else needs a
policy change.


## Triggers
workflow_dispatch, schedule, or a maintainer-only label. Instructions come
from files on main. Issue, PR, and log text is data, never instructions.


## Credentials
Agent jobs: contents: read, no deploy or production credentials, a dedicated
model key with a spend limit. Write access: a separate job with no model,
using the CI agent GitHub App (contents and pull requests only, no workflows).


## Limits
Pinned CLI, action, and model versions. timeout-minutes 30, one concurrent
run per spec, a turn and budget cap on every run, two repair attempts.


## Output
A draft pull request with the evidence bundle. Agents never merge,
approve, deploy, or mark a pull request ready.


## Protected paths
.github/, CODEOWNERS, .claude/, .codex/, .cursor/, AGENTS.md, CLAUDE.md,
deploy configuration, and test oracles. Changes here are written by people.


## Audit
Each run links trigger, spec, run log, commit, checks, approver, and release.
Monthly: accepted rate, revert rate, blocked protected-path attempts, and
cost per accepted pull request.

Copy-paste prompts for server-side agent runs

Section titled “Copy-paste prompts for server-side agent runs”

Commit the first prompt as .github/agent-prompts/implement-spec.md, which both workflows read. The other two run in the same job pattern with a different prompt file.

Replace SLUG in the third prompt with the change directory name.

How do you prove the CI loop is safe?

Section titled “How do you prove the CI loop is safe?”

You prove the boundaries with tests of the pipeline, not by reading what the agent wrote. Run these checks when you introduce the workflow and after every change to it:

TestHow to run itExpected result
Untrusted input pathDispatch with spec: ../README.mdThe agent job fails at the spec check
Agent edits CIMerge a test spec that asks for a change to .github/workflows/ci.ymlopen-draft-pr fails on the protected-path check
Agent cannot pushLook up the agent job’s token permissions in the run logOnly contents: read
Runaway runSet --max-budget-usd 0.5 (or a one-minute timeout-minutes) on a test specThe run stops at the cap and the run log records why
Gates run on agent PRsOpen a draft PR from a dispatched runEvery required check starts and reports
Human authorityMark the PR ready and try to merge it with the app token, without a code-owner approvalBranch protection refuses the merge

For ongoing evidence, review four numbers each month from the run logs and pull requests: the share of agent pull requests merged, the share reverted within 30 days, the count of blocked protected-path attempts, and the model cost per merged pull request. Define them once on the AI metrics panel and cost them with the cost per accepted change method. A rising revert rate means review has become a rubber stamp; rising blocked attempts mean the prompt or the task class is wrong.

What breaks when agents run inside CI/CD?

Section titled “What breaks when agents run inside CI/CD?”

The agent edits the gate to make itself green. A model asked to “make CI pass” may skip a test or relax a threshold. Recovery: the protected-path check, extended with your test-oracle paths, plus code-owner review on .github/ and the test oracles, and the rules on protecting the test oracle. Revert any merged gate change and re-run the affected pull requests.

Untrusted text reaches a job with write access. This is the Clinejection chain above. Recovery: move the task to a read-only job, rotate every secret the job could reach, and clear the Actions caches, because the reported chain poisoned the cache to reach later jobs. Handle it as an incident using the agent incident runbook.

A token is broader than the task. The July 2025 compromise of the Amazon Q Developer extension for VS Code (CVE-2025-8217, AWS advisory published 2025-07-26) traced its root cause to an improperly scoped GitHub token. Recovery: explicit permissions: on every job, permissions: {} at the top of the workflow, and a GitHub App per purpose; see agent identity and secrets.

The draft PR shows no checks. Pull requests opened with GITHUB_TOKEN do not trigger workflows. Recovery: open them with the GitHub App token, as the reference job does.

An upgrade changes behaviour silently. An unpinned CLI, action, or model default changes what the agent does overnight. Recovery: pin the CLI version and model ID, pin third-party actions to a reviewed tag or commit SHA, and upgrade through a pull request that runs the self-test.

Reviewers approve green agent PRs without reading the evidence. The loop is then automation that outruns authority. Recovery: require the evidence bundle check, sample one merged agent pull request per week for a full review, and cap the number of open agent PRs per reviewer, as on the team parallelism page.

Where to go next with CI agents

Section titled “Where to go next with CI agents”

Place this loop among the other controls, then configure it per tool.

Governance and autonomy Which controls advise and which hold authority, by risk class.
Unattended agent runs Recurring work that runs with nobody watching.
Headless agents in CI The headless modes of each agent, compared.
Claude Code in CI/CD The Claude Code action and headless runs in detail.
Codex in CI/CD codex exec and the Codex GitHub Action in detail.
Cursor in CI/CD Cloud Agents, Automations, and Bugbot in a pipeline.
CTO Scorecard answer key Every scorecard question and its canonical page.

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings