Evaluation dataset
Also known as Eval dataset, Evaluation set
Definition
An evaluation dataset is a collection of cases used to assess a system against defined criteria. For an AI application, it can include inputs, expected behavior, reference answers, grading information, and the context needed to reproduce each case.
The cases define what the result means
A score is only informative about the cases and criteria behind it. If a dataset contains short formatting tasks, strong results do not establish that the same system can diagnose a complex repository bug.
Choose cases that represent the workload you intend to support. Include difficult, ambiguous, and failure-prone examples in addition to common requests. Record how the cases were selected so the resulting score has a clear interpretation.
Make the expected behavior inspectable
For an illustrative coding evaluation, a case might include a repository revision, a task description, permitted tools, and tests that check the requested behavior. The final answer text alone may be insufficient to judge success.
Some tasks have a single verifiable answer. Others require a rubric or human judgment. Reference answers can help, but they should not penalize a different valid solution solely because its wording or implementation differs.
Protect the comparison
Keep a held-out set when tuning prompts or routing policies. Repeatedly changing the system to improve the same small dataset can produce an optimistic result that does not transfer to new cases.
Version the dataset and its graders. A changed test, unavailable dependency, or altered repository can change the score without a model change. Preserve enough information to reproduce the run, and report important exclusions rather than quietly dropping inconvenient failures.
How this relates to Weave
An evaluation dataset lets a team test whether Weave Router's model choices work for its actual coding tasks. Include the tools, repository context, and grading rules used by the agent. Public benchmarks offer useful evidence, but your own workload can expose different requirements and failure patterns.
Explore Router