Tokenization
Also known as LLM tokenization, Tokenizer
Definition
Tokenization is the process of converting text into the token units a language model receives and generates. A token can represent a word, part of a word, punctuation, or another piece of text, depending on the tokenizer.
Text is not counted in words
Language models operate on tokens, not a universal word counter. Common words may map to one token, while unusual identifiers, code, punctuation, or multilingual text can map to several.
Different model families can tokenize the same prompt differently. A repository with long names and code may therefore consume a different number of tokens after a model change, even when the visible input is identical.
Why the count matters
Token counts affect context-window capacity and often contribute to input and output charges. An illustrative coding request that includes a repeated stack trace may use more context than the question suggests. The generated answer adds its own tokens.
Use the tokenizer and billing rules associated with the model you are evaluating. Do not estimate a production bill by multiplying character count by a fixed ratio without checking the actual distribution.
Token counts are not quality
A shorter tokenization can reduce cost or leave more room in context. It does not prove that a model understands the request better. Preserve the information needed for the task, and compare models on successful outcomes, latency, and total cost.
How this relates to Weave
Tokenization helps explain why the same engineering prompt can have different costs and context usage across routing candidates. Weave Router evaluations should measure real requests with the tokenizer and provider accounting used in production.
Explore Router