Benchmark contamination
Also known as Data contamination, Test set contamination
Definition
Benchmark contamination occurs when evaluation examples, answers, or close duplicates appear in a model's training data or development process. The model may then recall the benchmark instead of demonstrating transferable capability.
How contamination happens
Public benchmark questions can enter pretraining corpora, fine-tuning sets, prompt libraries, or manual prompt development. Near duplicates can also leak across datasets. Contamination is difficult to prove from a score alone, especially when a benchmark has been widely circulated.
Reduce the risk
Keep a private holdout set, use newly collected tasks, and test paraphrases or fresh examples. Document what data was visible during prompt and model development. A strong evaluation combines public comparability with private evidence that reflects the product's actual work.
Interpret scores carefully
A rising benchmark score may reflect capability, memorization, or improved test familiarity. Weave can preserve evaluation lineage and production outcomes together, helping teams see whether an offline gain predicts useful completed tasks.
Keep a short record of dataset provenance, collection dates, and transformations. When a benchmark is refreshed, report the change rather than presenting the new score as a perfectly comparable continuation. That documentation makes model decisions easier to audit.
How this relates to Weave
Weave helps teams keep evaluation inputs, model versions, and run timestamps connected. That record supports audits when a benchmark result changes and makes it easier to compare a held-out task set with production examples.
Explore Token intelligence