LLM inference
Also known as Language model inference
Definition
LLM inference is the process of running a trained language model on an input to produce an output. For a text-generating model, it typically involves processing the input context and generating additional tokens according to a decoding strategy.
A response has more than one stage
The serving system first processes the request's context. For many autoregressive language models, generation then proceeds token by token, with each new token influenced by the preceding context and generated output.
A token is not necessarily a whole word. Tokenization depends on the model, so identical text can produce different token counts across model families.
Inference normally uses existing model weights. It is distinct from training, which updates model parameters. Including examples in a prompt can influence a response without changing those weights.
Why request shape matters
An illustrative coding request might contain a short question and several large files. Another might contain a small input but ask for a long implementation. Both can consume substantial resources for different reasons.
Input length, generated length, model architecture, hardware, batching, and caching all affect performance. Network and queueing delays also influence the time a user observes, even when the model's computation is unchanged.
Measure the task, not just the call
A faster response is useful only if it meets the application's requirements. If a quick answer causes the agent to repeat its work, the complete workflow may become slower or more expensive.
Compare inference options using representative requests and a consistent outcome check. Record time to first output separately from the time to finish the response. For an agent, also include tool execution and any additional model calls needed to complete the task.
How this relates to Weave
Weave Router chooses models for inference requests within coding-agent workflows. Understanding inference helps explain why the least expensive model per token is not always the least expensive way to finish a task. Input processing, generated output, and repeated attempts all contribute to the result.
Explore Router