# When a new model ships: a 48-hour evaluation playbook

URL: https://developertoolkit.ai/en/ecosystem/models/new-model-playbook/
Description: A 48-hour procedure for a new coding model: pin today's setup, re-run your evals, compare quality, cost and latency, re-tune effort, and roll out per loop.

The 48-hour new-model playbook is a fixed procedure a team runs whenever a vendor ships a coding model: pin the current configuration, re-run the team's own eval set against the candidate, compare accepted-task rate, cost per accepted task and latency, re-tune effort before prompts, then roll out one agent loop at a time behind written rollback criteria.

Claude Opus 5.5 lands on a Tuesday. By Wednesday, half your team is on it because Claude Code auto-updated, and another lead wants GPT-6 Astra because of a launch chart. This page is for the tech lead who decides by Thursday, the CTO who signs off, and the developer who runs the evals.

## What the 48-hour playbook gives your team

- **A two-day timeline with an owner per step**, and pinning commands for Claude Code, Codex and Cursor, so a release reaches your team only after you decide.
- **A triage checklist and a decision table** that turn release notes and eval results into "switch", "stay" or "switch for some loops".
- **Rollback criteria per agent loop** and three copy-paste prompts for release day.

This page assumes you already have an internal eval set. If you do not, build one first with [the benchmarks and evals guide](/en/ecosystem/models/benchmarks/), which sizes a first set at about 30 tasks; this playbook reuses its tasks and commands. Current model names, prices and defaults live on [the model comparison hub](/en/ecosystem/models/), and this page does not repeat them.

## Why a new model is recurring work, not a one-off event

In 22 days of September 2026, four vendors shipped seven models that coding teams could pick up, and two tools changed their default model on their own:

| Release date | Vendor | Model | What changed in the tools |
| ---- | ------ | ----- | ------------------------- |
| 2026-09-01 | Anthropic | Claude Fable 5.1 | Available in Claude Code from v2.1.257; never a default |
| 2026-09-02 | Google | Gemini 3.8 Flash | Generally available in the Gemini API on 2026-09-02; added to GitHub Copilot on 2026-09-03 |
| 2026-09-03 | OpenAI | GPT-6 Astra | Codex's bundled default from CLI 0.153.4 on 2026-09-04, at `low` effort |
| September 2026 (xAI's release notes); 2026-09-21 in Cursor and GitHub Copilot | xAI | Grok 4.7 | Added to GitHub Copilot on 2026-09-21; Cursor's release notes list it the week of 2026-09-21 and Cursor calls it its flagship model |
| 2026-09-22 | OpenAI | GPT-6 Sol, GPT-6 Luna | In the Codex model picker from CLI 0.156.1 on 2026-09-23 |
| 2026-09-22 | Anthropic | Claude Opus 5.5 | Claude Code default from v2.1.280 on the `latest` channel, at `medium` effort |

Two of those rows changed what agents ran without anyone deciding it. **A model change is a production change, and your tools can make it for you.**

Then it happened again. These releases followed between 2026-09-28 and 2026-10-12:

| Release date | Vendor | Model | What changed in the tools |
| ---- | ------ | ----- | ------------------------- |
| 2026-09-28 | Anthropic | Claude Sonnet 5.5 | The `sonnet` alias on the Anthropic API from Claude Code v2.1.284, so a config that said `sonnet` changed model with an update; on both release channels on 2026-10-17 (`stable` is v2.1.292) |
| 2026-09-29 | OpenAI | GPT-6.1 Sol | Codex's bundled default from CLI 0.159.1 on 2026-09-29, at `low` effort, replacing GPT-6 Astra, which moved to priority 2 |
| 2026-09-30 | Google | Gemini 4 Argon (announced) | Limited to trusted cyber defenders through Google's Fairwind Program; no public API, so nothing to evaluate yet |
| 2026-10-07 | Anthropic | Claude Haiku 5.5 | The `haiku` alias on the Anthropic API from Claude Code v2.1.293, which only the `latest` channel had on 2026-10-12 |
| 2026-10-12 | Meta | Muse Wonder | On the Meta Model API and in Muse Code; Muse Wonder Full is available only to Muse Code subscribers |

## The 48-hour plan at a glance

| Window | Step | Output | Owner |
| ------ | ---- | ------ | ----- |
| Hour 0–2 | Freeze the current configuration and triage the release | Pinned config commit; triage notes | Eval owner |
| Hour 2–12 | Re-run the eval set: baseline versus candidate | Two Harbor job directories | Eval owner |
| Hour 12–24 | Compare the four numbers; re-tune effort, then instructions | Comparison table; tuned candidate | Eval owner, one developer |
| Hour 24–40 | Canary the candidate in one agent loop | Canary metrics against rollback criteria | Tech lead |
| Hour 40–48 | Decide per loop and record it | Updated routing record; rollback plan | Tech lead approves; CTO for a vendor change |

[The operating model](/en/org/operating-model/) explains who owns the eval suite across teams. Forty-eight hours is a target, not a deadline: if the evals show a tie, the answer is "stay and re-test at the next release".

## Run the playbook step by step

1. **Freeze the current configuration (hour 0).** Pin the model by its full ID, pin the effort, and pin the tool version wherever your tool updates itself. Aliases move: the Claude Code 2.1.296 help text describes `fable`, `opus` and `sonnet` as aliases "for the latest model" (checked 2026-10-12), so a config that says `opus` changed model on 2026-09-22, and one that says `sonnet` on 2026-09-28 (v2.1.284), without a commit. Commit the pins (CI workflow, `.claude/settings.json`) so that rollback later is a `git revert`.

2. **Triage the release notes (hour 0–2).** Read the vendor's model page and the tool's changelog, not the launch post, for five kinds of change:
   - **Default effort shifts.** Claude Opus 5.5 defaults to `medium` effort in Claude Code and on the API, while Opus 5 defaulted to `high`. Anthropic's effort docs say a request that omits `effort` "runs one level lower than it did on Claude Opus 5". Sonnet 5.5 and Haiku 5.5 also default to `medium` in Claude Code, but Sonnet 5.5 defaults to `high` on the API. In Codex, GPT-6.1 Sol and GPT-6 Astra default to `low` and GPT-6 Sol and Luna to `medium`.
   - **Removed or rejected parameters.** On Claude 4.7 and later models, a non-default `temperature`, `top_p` or `top_k` returns a 400 error. Opus 5.5 also rejects disabling thinking and forced tool use. Sonnet 5.5 rejects `thinking: {"type": "disabled"}` (use `between_tools`) and forced tool use, and Haiku 5.5 rejects manual `budget_tokens` and assistant prefill. GPT-6 Astra has no `none` effort, GPT-6.1 Sol supports neither `none` nor `minimal`, and OpenAI's upgrade guide says to remove `temperature` and `top_p`.
   - **Token accounting.** Anthropic's pricing page says Claude 4.7 and later models use a tokenizer that produces about 30% more tokens for the same text, and Haiku 5.5 counts the same text as about 30% more tokens than Haiku 4.5. A lower price per token can still mean a higher cost per task, and Haiku 5.5's price depends on prompt length (over 100,000 tokens the whole request moves to the higher tier).
   - **Data and billing terms.** Claude Fable 5.1 is a Covered Model with data retention by default and may bill to usage credits, depending on plan. Check them against your data rules first.
   - **Retirement of your rollback target.** GPT-5.4 left Codex's bundled catalog in CLI 0.158.0 and migrates sessions to GPT-6 Sol, GPT-5.5 retires from Codex on 2026-10-14, and Claude Sonnet 4.5 retires on 2026-11-30 on Anthropic-operated platforms. Before you plan a rollback, confirm on [the model comparison hub](/en/ecosystem/models/) that the model you would roll back to stays available.

3. **Re-run the eval set, baseline versus candidate (hour 2–12).** Same tasks, same instructions commit, same harness version; change only the model, with at least three attempts per task. With the Harbor setup from [the benchmarks and evals guide](/en/ecosystem/models/benchmarks/), that is two commands per tool, shown in the tabs below. Save a manifest with every job: model ID, effort, `claude --version` or `codex --version`, `harbor --version`, and the commit of your `CLAUDE.md`, `AGENTS.md` and rules. The go/no-go prompt reads it from `manifest.txt` in each job directory.

4. **Compare four numbers, not one (hour 12–18).** A pass rate alone picks the wrong model when the candidate wins by spending more. Compare:
   - **Accepted-task rate**: tasks whose verifier passed *and* whose patch a reviewer would merge unchanged, checked on a sample of 10 patches by a human or a calibrated judge ([model-graded checks](/en/workflows/test/model-graded-checks/) shows how to calibrate one). Record the verdicts as `reviews.csv` (task, accepted yes or no) in each job directory, so the go/no-go prompt can use them.
   - **Cost per accepted task**: total tokens or dollars for the run divided by accepted tasks. Claude Code's `--output-format json` result carries `total_cost_usd` and `duration_ms`; `codex exec --json` emits token usage on each `turn.completed` event.
   - **Latency**: median wall time per task, which decides interactive loops.
   - **Per-task flips**: tasks the candidate newly passes and tasks it newly fails. Six gained and four lost is a different trade from two gained.

   On 30 tasks × 3 attempts, the 95% interval on one pass rate is about ±10 points, but you are comparing two rates, so the interval on the difference is roughly ±14 points. Correlated attempts on one task widen it further, because they are not 90 independent trials. Compute it with a paired comparison per task, for example by bootstrapping over tasks, and treat any difference inside it as a tie. Decide ties on cost, latency or reviewer acceptance.

5. **Re-tune effort before you touch instructions (hour 18–24).** Run the candidate at its default effort, one level above and one level below; each tab below shows the Harbor command for the sweep. The Claude Code docs claim "Opus 5.5 at `medium` matches or exceeds Opus 5 at `high`"; the sweep checks that on your tasks. Only then audit your agent instructions for old-model workarounds (the second prompt below), remove one at a time, and re-run the tasks it affects. Never change model, effort and instructions in the same run, or you cannot tell which change moved the score.

6. **Canary the candidate in one loop (hour 24–40).** An agent loop is one place where an agent runs with its own trigger and reviewer: interactive sessions, a CI fix job, a review bot, a scheduled loop, a cloud agent. Roll out in that order, most supervised first, starting with volunteers in interactive sessions. Write the [rollback criteria](#rollback-criteria-to-write-before-the-rollout) into the rollout ticket before you flip the first pin.

7. **Decide per loop and record it (hour 40–48).** Use the decision table below; the answer is often "switch for some loops". Record the decision, the evidence and the rollback target in your routing record, as described in [model routing](/en/developer-scorecard-guide/model-routing/), and add any task the candidate failed in an interesting way to the eval set.

## Pin and evaluate the candidate in Claude Code, Codex and Cursor

The steps are the same for every tool; what differs is where the pin lives and what can move it without a commit. Harbor is installed once with `uv tool install harbor`; load each API key from your secret manager into the environment and never type it on a command line.

**Claude Code**

**Pin the baseline model and the channel.** In the project's `.claude/settings.json`, set today's model by full ID. In this example the baseline is Claude Opus 5 and the candidate is Opus 5.5:

```json title=".claude/settings.json"
{
  "model": "claude-opus-5"
}
```

For the tool version, set `"autoUpdatesChannel": "stable"` for anyone who should not receive a model default change on release day. On 2026-09-26 `stable` was v2.1.274, which runs Opus 5 but not Opus 5.5 (that needs v2.1.280+). Only canary volunteers move to the `latest` channel and switch their pin to `claude-opus-5-5`. The channel only delays a model, and by days: `stable` reached v2.1.292 and on 2026-10-17 runs Opus 5.5 (v2.1.280) and Sonnet 5.5 (v2.1.284) but not Haiku 5.5 (v2.1.293, `latest` only), so the pin, not the channel, is what keeps a candidate out. To keep a candidate out of reach until it is approved, see [managed policy](/en/org/managed-policy/) for `availableModels`, `enforceAvailableModels` and, from v2.1.283 (both release channels on 2026-10-12), `deniedModels`.

**Pin CI jobs on the command line**, which outranks every settings file except managed policy. A `claude -p` run starts in Manual mode, so grant only the edits and the one test command the job needs, and keep the prompt before `--allowedTools`, which takes a list:

```bash
claude -p "Run the failing test in src/billing and fix the cause, not the test." \
  --model claude-opus-5 --effort high --output-format json \
  --permission-mode acceptEdits --allowedTools "Bash(npm test *)"
```

**Run the eval pair** from the terminal, changing only the model:

```bash
harbor run -p evals/tasks -a claude-code -m anthropic/claude-opus-5 -k 3 -n 4
harbor run -p evals/tasks -a claude-code -m anthropic/claude-opus-5-5 -k 3 -n 4
```

**Write a manifest** into each job directory Harbor created, filling in the model and effort of that run:

```bash
{ echo "model=claude-opus-5-5"; echo "effort=medium"; claude --version; harbor --version; \
  echo "instructions=$(git rev-parse HEAD)"; } > jobs/JOB_DIR/manifest.txt
```

**Sweep effort** through Harbor's agent option `reasoning_effort`, which Harbor 0.23.0 passes to Claude Code as `--effort`:

```bash
harbor run -p evals/tasks -a claude-code -m anthropic/claude-opus-5-5 -k 3 -n 4 \
  --ak reasoning_effort=high
```

Claude Code v2.1.296 accepts `low`, `medium`, `high`, `xhigh` and `max` for `--effort` (`claude --help`, checked 2026-10-12); in a session, use `/effort`.

**Codex**

**Pin the model and effort in config.** Codex 0.154.0 changed fresh sessions to "respect server model defaults unless explicitly overridden", so for a signed-in ChatGPT account the effective default is decided server-side. An explicit `model` is the only reliable pin:

```toml title="~/.codex/config.toml"
model = "gpt-6.1-sol"
model_reasoning_effort = "low"
```

That file lives in each developer's home directory, outside git, so distribute it as a shared team dotfile or each pin drifts on its own. Put the pin that rollback depends on in the CI workflow, committed with the code (the `:workspace` permission profile, beta, lets the run write its fix):

```bash
codex exec --json -m gpt-6.1-sol -c model_reasoning_effort=low \
  -c default_permissions=':workspace' \
  "Run the failing test in src/billing and fix the cause, not the test."
```

Keep the candidate one flag away: `--profile NAME` (`-p`) layers `$CODEX_HOME/NAME.config.toml` on top of the defaults:

```toml title="~/.codex/candidate.config.toml"
model = "gpt-6-astra"
model_reasoning_effort = "medium"
```

```bash
codex -p candidate exec --json -c default_permissions=':workspace' \
  "Run the failing test in src/billing and fix the cause, not the test."
```

Pin the CLI in CI with an exact package version, for example `npm install -g @openai/codex@0.162.1`, so a new release cannot change the bundled default mid-evaluation: CLI 0.159.1 changed it from GPT-6 Astra to GPT-6.1 Sol on 2026-09-29. Codex 0.157.0 added migration prompts to GPT-6 Sol or Luna; decline them until the evaluation is done.

**Run the eval pair:**

```bash
harbor run -p evals/tasks -a codex -m openai/gpt-6.1-sol -k 3 -n 4
harbor run -p evals/tasks -a codex -m openai/gpt-6-astra -k 3 -n 4
```

**Write a manifest** into each job directory:

```bash
{ echo "model=gpt-6-astra"; echo "effort=medium"; codex --version; harbor --version; \
  echo "instructions=$(git rev-parse HEAD)"; } > jobs/JOB_DIR/manifest.txt
```

**Sweep effort** with the same agent option; Harbor 0.23.0 turns it into `-c model_reasoning_effort=high` on the `codex exec` command line:

```bash
harbor run -p evals/tasks -a codex -m openai/gpt-6-astra -k 3 -n 4 \
  --ak reasoning_effort=high
```

Codex effort levels run from `low` to `max`, plus `ultra` ("Maximum reasoning with automatic task delegation"), which GPT-6.1 Sol and GPT-6 Astra offer and GPT-6 Luna does not.

**Cursor**

**Pin by selection, and verify in your own account.** Cursor names no single default model, and Auto routes each request (checked 2026-10-12). Choose the model explicitly in the model picker, not an automatic choice, and name it in your team's rules. [The models hub](/en/ecosystem/models/) has the per-model rates Cursor published on 2026-10-12.

**Run the eval pair** with Harbor's `cursor-cli` agent, which requires `CURSOR_API_KEY` and runs the Cursor CLI in print mode:

```bash
harbor run -p evals/tasks -a cursor-cli -m cursor/BASELINE_MODEL -k 3 -n 4
harbor run -p evals/tasks -a cursor-cli -m cursor/CANDIDATE_MODEL -k 3 -n 4
```

Replace `BASELINE_MODEL` and `CANDIDATE_MODEL` with model IDs your Cursor account lists, and write a `manifest.txt` into each job directory as in the other tabs. Never carry a score across from Claude Code or Codex.

**Harbor runs agents with approvals off**

Harbor's Codex agent passes `--dangerously-bypass-approvals-and-sandbox` and its Cursor agent passes `--yolo`. That is acceptable only because each trial runs in a disposable container. Put nothing in the task image except the repository snapshot, and no credential except the model key. Restrict the container's outbound network to the model API and the hosts the agent installer downloads from, so a prompt-injected agent has nowhere to send what it reads. Harbor 0.23.0 has no top-level `--effort`; pass effort as an agent option (`--ak reasoning_effort=…`) and confirm what your version accepts with `harbor agent schema <agent>`.

## How to decide from the eval results

Read the four numbers from step 4 against this table. "Noise" means the interval on the difference between the two runs: roughly ±14 points for two runs of 30 tasks × 3 attempts, and wider with correlated attempts.

| What your eval shows | Decision | Next action |
| -------------------- | -------- | ----------- |
| Candidate's accepted-task rate is higher by more than the noise, at similar cost | Switch, loop by loop | Canary in interactive sessions first |
| Accepted-task rates tie; candidate is clearly cheaper per accepted task or faster | Switch the loops where cost or latency dominates | High-volume CI loops first; keep the baseline on long-horizon work |
| Candidate wins only at a higher effort that costs more per accepted task | Switch only the loops that need it | Record the effort per loop in the routing record |
| Everything ties | Stay | Record "no change" with the job IDs; re-test at the next release |
| Candidate loses by more than the noise, or fails a task class you depend on | Stay and keep the pin | Add the failing tasks to the eval set |

A launch chart is not a row in this table. Anthropic itself writes that "at these levels of capability we've found that benchmark margins have become a less reliable guide to real-world differences", and [the benchmarks guide](/en/ecosystem/models/benchmarks/) explains why a public score belongs to a model and harness pair, not to the model.

## Rollback criteria to write before the rollout

Write the thresholds into the rollout ticket before the first loop switches, so nobody negotiates them mid-incident. The numbers below are starting points for a team running 20 or more agent tasks a day.

| Signal in the canary loop | Start with this threshold | Action |
| ------------------------- | ------------------------- | ------ |
| Accepted-task rate (verifier pass and reviewer acceptance) | Below a baseline rate measured on at least a few hundred historical tasks by more than the canary's own interval: about ±22 points at 20 canary tasks, ±14 at 50. With a smaller baseline, use the interval on the difference (about ±31 and ±20) and bootstrap as in step 4 | Revert the pin in that loop |
| Cost per accepted task | More than 25% above the baseline for three working days | Lower effort one level, re-measure; revert if still over |
| Median wall time per task in interactive sessions | More than 50% above the baseline | Revert the interactive loop; keep batch loops if they pass |
| Reviewer rework rate on agent pull requests | More than 10 points above the baseline | Sample 10 pull requests; revert if the rework is model-related |
| Security-gate failure, secret exposure or data-policy breach linked to the change | One occurrence | Revert immediately and open an incident |

Rollback is a revert of the pin commit (`.claude/settings.json` or the CI workflow; a `~/.codex/config.toml` pin is redistributed, not reverted), which any engineer on call can make without a meeting. [The rollback pipeline guide](/en/developer-scorecard-guide/rollback-pipeline/) covers the same pattern for code; [progressive delivery](/en/workflows/ship/progressive-delivery/) covers staged rollout mechanics; and [agent cost tracking](/en/ecosystem/observability/agent-cost-tracking/) covers where the cost signal comes from.

## How the switch is proven without reading every diff

Nobody reads the candidate's code line by line. The evidence is verifier-graded evals on your own tasks, a reviewer-acceptance sample of 10 patches per configuration (the only human reading), unchanged CI gates during the canary, and a decision record with job IDs so anyone can re-run it.

**Who signs off:** the eval owner publishes the comparison; the tech lead approves each loop's switch and owns the rollback; the CTO approves a change of vendor or a change that affects data terms, such as moving to a Covered Model. [Continuous evals](/en/developer-scorecard-guide/continuous-evals/) runs the same eval set on every change to your agent instructions, so the next release starts from a fresh baseline.

## Copy-paste prompts for release day

**Copy-paste prompt: triage a model release against our setup**

```text
A new model was released. Read the vendor's model page and the tool changelog
at the URLs at the end of this message, then audit this repository against them.

1. List every change in the release that can affect us, grouped as: default effort,
   removed or rejected API parameters, tool-use behaviour, token accounting,
   data retention or billing terms, and retirement of older models.
2. Search this repository for code and config that each change touches:
   model IDs and aliases (opus, sonnet, haiku, fable, gpt-6-*, gpt-6.1-*), temperature, top_p, top_k,
   budget_tokens, forced tool_choice, effort settings, model_reasoning_effort,
   and any CI workflow that calls claude -p or codex exec.
3. Output a table: change, file and line affected, risk (breaks / costs more /
   no effect), and the smallest fix. Do not edit any file.
4. Name the model we would roll back to and whether the release notes change
   its availability.

Model page URL:
Changelog URL:
```

**Copy-paste prompt: find old-model workarounds in our agent instructions**

```text
Read CLAUDE.md, AGENTS.md, every file under .claude/, .codex/, .cursor/rules
and .agents/skills in this repository. Find instructions that exist to work
around a weakness of an older model rather than to state a project rule:
repeated warnings, "always re-read the file before editing", step-by-step
scaffolding for routine tasks, forced verbosity or terseness, and rules that
duplicate what a test or linter already enforces.

For each one, report: the file and line, the probable weakness it compensates
for, which eval tasks it could affect, and a proposed removal. Rank by how
much context each one costs. Do not edit anything; I will remove them one at a
time and re-run the affected eval tasks after each removal.
```

**Copy-paste prompt: go/no-go from two eval runs**

```text
Compare the two Harbor job directories named below: BASELINE and CANDIDATE.
Each job directory has a manifest.txt with model ID, effort, tool version and
instructions commit; stop and tell me if the manifests differ in anything
other than model or effort.

Report for each job: pass rate, median wall time per task, and total tokens
or cost if the logs contain them. Then list per task: passes out of attempts
in each job, and the tasks that flipped. Estimate a 95% interval on the
difference in pass rate by bootstrapping over tasks (resample tasks, keep
each task's attempts together); do not treat attempts as independent.
If a job directory has a reviews.csv listing reviewer-accepted tasks, report
accepted-task rate and cost per accepted task as well, and apply the table
to those; otherwise say that the decision covers the verifier half only.

Apply this decision table and name one row:
- candidate higher by more than the interval on the difference, at similar cost: switch loop by loop
- tie, candidate clearly cheaper or faster: switch cost- or latency-bound loops
- candidate wins only at a higher effort that costs more: switch only loops that need it
- tie on everything: stay
- candidate lower by more than the interval on the difference: stay and keep the pin

Read the files only; do not re-run anything.

BASELINE: jobs/
CANDIDATE: jobs/
```

## What goes wrong when you evaluate a new model?

- **The tool upgraded before you evaluated.** Symptom: sessions already run the new model on release day. Recovery: pin the full model ID (`claude-opus-5`) in `.claude/settings.json` and CI, which works on any version, and set `model` explicitly in Codex. Prevention for the next release: move everyone outside the canary to the `stable` channel.
- **The default effort dropped and nobody noticed.** Symptom: "the new model is lazier" on hard tasks. Recovery: compare at matched effort as well as at each model's default, and set effort explicitly in CI.
- **Cheaper per token, dearer per task.** Symptom: the invoice rises after a switch to a lower-priced model. Recovery: compare cost per accepted task, which captures tokenizer changes, extra turns and retries.
- **The eval set is saturated.** Symptom: baseline and candidate both pass nearly everything. Recovery: add recent hard tasks and every escaped agent defect; a set that cannot fail cannot choose.
- **Launch-week capacity.** Symptom: timeouts and rate-limit errors skew latency and pass rates. Recovery: re-run failed trials after 24 hours before you conclude anything, and use `--fallback-model` in Claude Code headless jobs.
- **The rollback target is gone.** Symptom: reverting the pin fails because the old model retired. Recovery: check retirement in step 2 and keep a second approved model in the routing record.

## Where to go next after a model evaluation

- [Reading benchmarks and building your own evals](/en/ecosystem/models/benchmarks/) — The prerequisite: build the Harbor eval set this playbook re-runs.
- [Model comparison hub](/en/ecosystem/models/) — Current model names, prices, defaults and retirement dates per tool.
- [Model routing](/en/developer-scorecard-guide/model-routing/) — Record the decision per task class and loop, with its evidence.
- [Continuous evals](/en/developer-scorecard-guide/continuous-evals/) — Run the same eval set on every change to your agent instructions.
- [Keeping current](/en/teams/keeping-current/) — A team cadence for vendor releases that does not turn into a weekly fire drill.
- [Open-weight models](/en/ecosystem/models/open-weight/) — Measure an open-weights candidate on the same eval set.
