Pass at k
Also known as pass@k, Pass-k
Definition
Pass at k is the probability that at least one of k generated samples solves an evaluation task. It measures the benefit of giving a model several attempts, rather than the reliability of its first answer.
What the score means
If a model receives k chances to solve a task, pass at k asks whether any attempt passes the evaluator. Pass at one is ordinary first-attempt success. Larger k values can show whether extra sampling unlocks solutions that the first response misses.
Why selection matters
Generating several candidates does not help unless the system has a way to select one. Tests, a verifier, a ranking model, or human review can provide that selection. Without a reliable selector, pass at k may overstate the quality a user experiences.
Track the tradeoff
Each additional sample consumes time and tokens. Record cost, completion latency, and the number of attempts alongside the score. Weave's trace and token views help teams decide whether extra sampling improves completed tasks enough to justify its spend.
How this relates to Weave
Weave can record repeated model attempts, test outcomes, latency, and token cost so teams can compare pass at k with the cost of generating k samples. A higher score is useful only when the application can afford and select among the attempts.
Explore Code intelligence