Hallucination

Why hallucinations occur

Language models generate likely continuations from patterns in their training and context. They do not automatically verify every claim against a current source. A model may therefore invent a citation, use a plausible but nonexistent API method, or fill a missing fact with an answer that sounds confident.

The risk depends on the task. A creative story can intentionally contain invented details. A support answer, code change, or medical explanation usually needs evidence and a clear way to express uncertainty. Longer prompts do not guarantee correctness, and a confident tone is not a reliability measure.

Measure the task, not just the text

Create an evaluation dataset that represents real requests and labels the claims or outcomes that matter. Use reference answers, retrieved evidence, executable tests, or human review as appropriate. Separate factual grounding from style, completeness, and task success because one score can hide a serious failure mode.

Retrieval augmented generation can give a model relevant evidence, but retrieval quality and citation behavior still need evaluation. Constrain output formats where possible, show sources to users, and give the model a way to say that evidence is missing.

Reduce risk in production

Monitor outputs by route and model version, sample difficult cases for review, and keep a record of the evidence used for important answers. Routing can send high-risk tasks to a stronger model or a workflow with verification. Weave helps teams connect these quality results to model usage, cost, and the engineering work required to improve the system.