Canonical: https://weaveos.com/glossary/tokens-per-second

[Home](https://weaveos.com/)/[Glossary](https://weaveos.com/glossary)/[Inference performance](https://weaveos.com/glossary/category/inference-performance)/Tokens per second

[Inference performance](https://weaveos.com/glossary/category/inference-performance)

# Tokens per second

Also known as Tokens per second, AI Tokens per second

By WeavePublished September 10, 2026Updated October 17, 20262 min read

## Definition

Tokens per second (TPS) is the rate at which an AI model generates output tokens. Per-request TPS and system TPS use different measurement scopes and should be compared only when their definitions align.

On this page

1. [Definition](https://weaveos.com/glossary/tokens-per-second#definition)
2. [What tokens per second means](https://weaveos.com/glossary/tokens-per-second#article-what-tokens-per-second-means)
3. [How to calculate and compare TPS](https://weaveos.com/glossary/tokens-per-second#article-how-to-calculate-and-compare-tps)
4. [Interpret TPS with other outcomes](https://weaveos.com/glossary/tokens-per-second#article-interpret-tps-with-other-outcomes)
5. [Higher TPS alone does not establish lower end-to-end latency, lower cost, or better task quality. Interpret it with time to first token, completion latency, reliability, cost, and task outcome under a representative workload. In a dated analysis of real agent requests, Weave reports output throughput with TTFT and total upstream latency, and cautions that each model sees a different traffic mix. Treat those results as observations of that workload, not as controlled claims about intrinsic model speed.](https://weaveos.com/glossary/tokens-per-second#article-higher-tps-alone-does-not-establish-lower-end-to-end-latency-lower-cost-or-better-task-quality-interpret-it-with-time-to-first-token-completion-latency-reliability-cost-and-task-outcome-under-a-representative-workload-in-a-dated-analysis-of-real-agent-requests-weave-reports-output-throughput-with-ttft-and-total-upstream-latency-and-cautions-that-each-model-sees-a-different-traffic-mix-treat-those-results-as-observations-of-that-workload-not-as-controlled-claims-about-intrinsic-model-speed)
6. [How this relates to Weave](https://weaveos.com/glossary/tokens-per-second#weave)
7. [Sources](https://weaveos.com/glossary/tokens-per-second#sources)
8. [Related terms](https://weaveos.com/glossary/tokens-per-second#related)

## What tokens per second means

Tokens per second (TPS) expresses output-token throughput, but per-request TPS and system TPS describe different scopes. NVIDIA AIPerf defines per-user TPS as an individual response's output-token count divided by its end-to-end latency. Its system TPS aggregates output tokens across concurrent requests over the benchmark interval, from the first request to the last response. System TPS can rise with concurrency even as individual requests take longer.

## How to calculate and compare TPS

For a per-request measurement, divide output tokens by that request's end-to-end latency. A response with 120 output tokens over four seconds of end-to-end latency measures 30 TPS under NVIDIA's per-user definition. Amazon Bedrock's CloudWatch example estimates OTPS for each five-minute period as `p50(OutputTokenCount) / (p50(InvocationLatency) - p50(TimeToFirstToken)) * 1,000`. Both latency metrics are in milliseconds, so multiplying by 1,000 converts tokens per millisecond to tokens per second. This uses separate p50 metrics, so it estimates generation throughput from aggregate percentiles rather than calculating each request's end-to-end TPS. State whether TTFT is included and whether the result is per request or system-wide before comparing values. Include the model, workload, output length, concurrency, batching, and benchmark window.

## Interpret TPS with other outcomes

## Higher TPS alone does not establish lower end-to-end latency, lower cost, or better task quality. Interpret it with time to first token, completion latency, reliability, cost, and task outcome under a representative workload. In a [dated analysis of real agent requests](https://weaveos.com/blog/we-measured-time-to-first-token-on-275-000-real-agent-requests.-the-benchmarks-lied.), Weave reports output throughput with TTFT and total upstream latency, and cautions that each model sees a different traffic mix. Treat those results as observations of that workload, not as controlled claims about intrinsic model speed.

## How this relates to Weave

A routing comparison should report tokens per second with the model, provider, workload, and request outcome. Weave's Router analysis reports output throughput alongside time to first token and total upstream latency for real agent requests; the Router does not make an inference runtime faster by itself.

[Explore Router](https://weaveos.com/products/router)

## Sources and further reading

1. [NVIDIA AIPerf metrics: Tokens Per Second](https://docs.nvidia.com/nim/benchmarking/llm/metrics#tokens-per-second)
2. [Amazon Bedrock output tokens per second (OTPS)](https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-otps.html)

## Related terms

### [Time to first token](https://weaveos.com/glossary/time-to-first-token)

[What time to first token measures in streaming AI responses and why it matters for user experience and routing.](https://weaveos.com/glossary/time-to-first-token)

### [Quantization](https://weaveos.com/glossary/quantization)

[What model quantization changes in an AI system, how lower precision can reduce memory use, and which quality and latency questions to test.](https://weaveos.com/glossary/quantization)

### [Inference throughput](https://weaveos.com/glossary/inference-throughput)

[How inference throughput measures model serving capacity, how tokens per second differs from request throughput, and why workload shape matters.](https://weaveos.com/glossary/inference-throughput)
