Tokens per second

What tokens per second means

Tokens per second (TPS) expresses output-token throughput, but per-request TPS and system TPS describe different scopes. NVIDIA AIPerf defines per-user TPS as an individual response's output-token count divided by its end-to-end latency. Its system TPS aggregates output tokens across concurrent requests over the benchmark interval, from the first request to the last response. System TPS can rise with concurrency even as individual requests take longer.

How to calculate and compare TPS

For a per-request measurement, divide output tokens by that request's end-to-end latency. A response with 120 output tokens over four seconds of end-to-end latency measures 30 TPS under NVIDIA's per-user definition. Amazon Bedrock's CloudWatch example estimates OTPS for each five-minute period as p50(OutputTokenCount) / (p50(InvocationLatency) - p50(TimeToFirstToken)) * 1,000. Both latency metrics are in milliseconds, so multiplying by 1,000 converts tokens per millisecond to tokens per second. This uses separate p50 metrics, so it estimates generation throughput from aggregate percentiles rather than calculating each request's end-to-end TPS. State whether TTFT is included and whether the result is per request or system-wide before comparing values. Include the model, workload, output length, concurrency, batching, and benchmark window.

Interpret TPS with other outcomes

Higher TPS alone does not establish lower end-to-end latency, lower cost, or better task quality. Interpret it with time to first token, completion latency, reliability, cost, and task outcome under a representative workload. In a dated analysis of real agent requests, Weave reports output throughput with TTFT and total upstream latency, and cautions that each model sees a different traffic mix. Treat those results as observations of that workload, not as controlled claims about intrinsic model speed.