How Weave Router benchmarks are run

What does this benchmark evaluate?

The public weave-bench tooling compares Codex CLI through Router’s /beta lane with the same client pointed at pinned model or metarouter controls. It covers SWE-Atlas Codebase QnA and Terminal-Bench 4.0. The reviewed harness explicitly excludes Claude Code harnesses.

This evaluates an agent configuration on a defined task set. It does not establish performance for every coding workflow, a subscription’s task capacity, or the financial ROI of a customer team. The coding-subscription evaluation guide covers that separate decision.

Which tasks, versions, and scoring are pinned?

The following are documented in the reviewed harness revision. Check the current dry-run output before a new experiment.

The repository records the dataset references and manifests. It also documents that published Atlas runs used Harbor 0.18.0 with a private patch, while the public reproduction tooling uses Harbor 0.22.0 with an equivalent agent path. That difference belongs in a reproduction report.

How do you inspect a run before executing it?

Follow the repository’s locked dependency setup first. Then inspect the plan from the benchmark environment:

weave-bench run terminal-bench-4 --arms router,luna --smoke --dry-run
weave-bench run atlas-qna --dry-run

The dry-run resolves the manifest, version pins, arm configurations, environment-variable names, and launcher arguments without network calls. It is not a scored trial. A real benchmark makes paid provider requests and requires the configured sandbox and credentials.

Before a real run, use weave-bench probe to record the serving Router build and routing package and verify the /beta acknowledgement. Keep that output with the results. A changed build or package can change the comparison even when the command stays the same.

How should attempts and uncertainty be reported?

The harness pairs trials by task and attempt. Keep both attempts, failed trials, verifier outputs, and exceptions. Distinguish task-mean pass rates from trial-level pass rates; their denominators differ.

Report paired differences with the uncertainty supplied by the analysis. A confidence interval spanning zero does not prove equivalence. Record the judge model for rubric-based evaluation and distinguish model failures from environment setup failures and provider refusals.

Reasoning effort also matters. The documented Atlas Astra control uses max, while other arms use high. Published Terminal-Bench launchers left effort at the client default; the public reproduction harness pins high. A rerun is therefore effort-pinned rather than launcher-exact. Preserve this distinction next to the result.

Where do the cost columns come from?

Router-billed fields require analytics access to the run’s organization or a saved export. If unavailable, the report leaves them blank. A missing cost must not become a zero or a modeled replacement under the same label.

Show judge costs, retries, smoke runs, and infrastructure separately from the reported comparison when relevant. Cache behavior and missing trial coverage can materially change a cost comparison.

What should accompany a published result?

Attach the source revision, run date, task manifest, probe output, arm settings, attempt counts, failures, scoring definition, paired analysis, and cost provenance. Include raw trial artifacts or an accessible export when permission allows. The public harness does not grant access to another organization’s private telemetry.

Download the benchmark reporting checklist (CSV) to record those fields. It is a blank reporting aid, not a dataset of benchmark results. Cite this methodology together with the exact source run rather than using a product headline as the experiment definition.