Top-p sampling
Also known as Nucleus sampling, LLM top-p
Definition
Top-p sampling, also called nucleus sampling, selects the next token from the smallest group whose cumulative probability reaches a chosen probability p. Lower p values restrict the candidate set, while higher values allow more candidates.
How nucleus sampling works
Suppose the model assigns probabilities to many possible next tokens. Top-p sorts those candidates by probability and keeps the smallest prefix whose combined probability reaches p. The next token is sampled from that set. The set can be large when the model is uncertain and small when one continuation dominates.
Top-p is different from top-k sampling, which always keeps a fixed number of candidates. It is also different from temperature, which reshapes the full probability distribution before sampling. Providers may expose both parameters, but combining them changes the distribution in ways that should be tested rather than assumed.
Pick settings through evaluation
A lower p can reduce surprising continuations but may make outputs repetitive or omit a useful option. A higher p can create more variety while increasing the chance of an irrelevant or unsupported answer. The right setting depends on the task, model, prompt, and output constraints.
For a fair comparison, hold the model and prompt constant, record the random seed when supported, and evaluate enough examples to see variability. Measure task success, factuality, format validity, and user preference separately. Store the chosen parameter with the request so a later investigation can reproduce the behavior.
How this relates to Weave
Weave makes generation settings part of an evaluation record, which helps teams compare top-p and temperature choices against quality, cost, and latency on representative engineering tasks.
Explore Router