Inference throughput
Also known as LLM throughput, Model throughput
Definition
Inference throughput is the amount of model inference work completed in a period of time. It may be expressed as requests per second, input tokens per second, output tokens per second, or another workload-specific measure.
Throughput depends on the unit
Requests per second describes how many calls a service completes. Tokens per second describes how quickly it processes or generates token material. Both can be useful, but they answer different questions.
An endpoint serving short completions may handle many requests while producing fewer total tokens than a system handling long responses. A benchmark should state whether it measures input tokens, output tokens, completed requests, or a combination.
Workload shape changes results
Batch size, context length, output length, concurrency, and model configuration all affect throughput. A single isolated request usually measures a different operating point from a busy service with many simultaneous users.
Report the conditions with the number. Include warm-up behavior, failure rate, queue time, and the latency percentiles users experienced. Otherwise a throughput improvement can hide longer waits or more rejected requests.
Capacity is not quality
Throughput answers how much work a serving system can process. It does not establish whether responses solve the task. Pair capacity measurements with an evaluation dataset and an outcome definition before changing a production route.
How this relates to Weave
When comparing routes in Weave Router, throughput helps describe serving capacity alongside task latency and completion quality. A high tokens-per-second result on short prompts may not predict the behavior of a long coding-agent session.
Explore Router