Agent Run Comparison

Specify what changes between runs

A run includes more than its model. Instructions, repository state, available tools, permissions, context limits, and the stopping rule can all affect the result. If the question concerns a routing policy, keep the surrounding agent harness stable. If the question concerns a complete product, compare the full configurations and identify their differences.

Start each attempt from the same repository snapshot and reset external state when it matters. Record model versions, configuration, task identifier, tool errors, and the evaluation version. A result without that context is difficult to reproduce or explain.

SWE-bench illustrates a concrete evaluation unit: a repository and an issue that require a resolving patch. Internal comparisons can use a different task collection, provided its acceptance criteria are equally explicit.

Compare outcomes and resources separately

Decide which checks establish success before running the experiment. Keep correctness, security requirements, and review acceptance distinct from token cost and elapsed time. A configuration that is inexpensive because it gives up early has not necessarily improved efficiency.

Anthropic's evaluation guide distinguishes individual trials from tasks and recommends multiple trials because outputs vary. Repeated attempts reveal instability that a single successful demonstration can hide. Report the number of tasks and trials with every aggregate.

A hypothetical paired evaluation

Consider 20 issue-fixing tasks, each attempted three times by configurations A and B. All 60 attempts per configuration use the same time limit and acceptance checks.

Configuration B succeeds more often in this illustrative sample, while A spends less per successful attempt. That ratio includes money spent on failed attempts. Neither result alone determines the preferred configuration: inspect which tasks failed and how much human repair they required.

Show results by task as well as the overall total. If one configuration succeeds consistently on simple changes but fails every security-sensitive task, its average hides an important deployment constraint. Describe any excluded attempts and the reason for exclusion.

Avoid misleading comparisons

Do not compare a single-attempt success rate with a metric that counts success after several retries. Also keep warm-cache and cold-cache runs separate when their cost or latency differs materially. If the intended workflow permits retries, include their complete resource cost.

A small evaluation can detect obvious regressions without proving a universal ranking. Tests may miss defects, tasks may resemble training data, and production work may require different tools. Use the comparison to choose a configuration for a defined workload, then inspect its behavior on representative real changes and repeat the evaluation when that configuration changes.