Evaluation metric
Also known as Model evaluation metric, Eval metric
Definition
An evaluation metric is a defined calculation used to summarize how a system performs against an evaluation criterion. The calculation can compare predictions with references, classify outcomes, measure latency or cost, or combine several signals.
Tie the metric to a decision
Accuracy can be useful for a classification task. It says less about an agent that must edit files, run tests, and preserve an existing interface. That workflow may need pass or fail checks, human review, execution time, and cost.
An illustrative evaluation reports a 90% answer score from a reference-based metric. If the grader rewards wording that resembles the reference, the score may not reflect whether the answer solves the user's problem.
Report the definition with the result
State the dataset, scoring rule, exclusions, aggregation, and confidence or variation. A score averaged across easy and difficult task classes can hide a failure on the cases that matter most.
Keep separate metrics visible when they represent different tradeoffs. Combining quality, cost, and latency into one index can support a product decision, but the underlying values are needed to understand why the index moved.
Test transfer, not just fit
When a prompt or router is tuned against one dataset, use held-out cases to check whether the improvement transfers. Re-run after model, tool, or repository changes. A metric is an instrument for learning about a system, not a permanent label for its quality.
How this relates to Weave
Weave Router benchmarks combine task success with cost and speed evidence. Choose metrics that reflect the coding-agent behavior you care about, and keep the grading rule visible so a lower cost or higher score is not mistaken for a universal improvement.
Explore Router