Evaluation harness
Also known as LLM evaluation harness, AI eval harness
Definition
An evaluation harness is the software and configuration that runs an evaluation consistently. It typically loads cases, invokes a system, applies grading rules, records metrics, and produces results that can be compared across versions.
A harness turns a test into a repeatable run
An evaluation harness defines how cases are selected, how the system is called, and how outcomes are graded. It may run a classifier, chat application, retrieval pipeline, or coding agent. The same structure can then be used after a prompt, model, tool, or router change.
The harness should keep the case inputs and grading configuration visible. If a run silently changes its dataset or evaluator, a new score is difficult to interpret.
Capture more than the final score
Record the system version, model and provider, route decision, tool calls, latency, token usage, grader output, and errors where they are relevant. Aggregate results by task type so a strong average does not hide a regression in a critical class.
Reference-based grading can be useful for some tasks. Rubrics, executable checks, and human review may be better for others. The harness should support the evidence the task requires rather than forcing every behavior into one metric.
Use held-out cases
Tune prompts and routing policies on one set, then check them on cases kept out of development. Repeat runs when randomness affects results and report variation. A harness supports disciplined comparison, but it does not make a weak dataset or grader representative by itself.
How this relates to Weave
W&B Weave provides evaluation workflows for comparing model and application behavior. A harness makes the run repeatable and records enough context to connect a score to the prompts, model, route, tools, and dataset that produced it.
Explore Engineering intelligence