Find a term
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
Terms beginning with T
12 termsTail latency
Tail latency is the serving concept concerned with tail latency during AI inference.
Inference performanceTensor fusion
Tensor fusion is the serving concept concerned with tensor fusion during AI inference.
Inference performanceTensor parallelism
Tensor parallelism is the serving concept concerned with tensor parallelism during AI inference.
Inference performanceThroughput latency tradeoff
Throughput latency tradeoff is the serving concept concerned with throughput latency tradeoff during AI inference.
Inference performanceThroughput per gpu
Throughput per gpu is the serving concept concerned with throughput per gpu during AI inference.
Inference performanceTime to first byte
Time to first byte is the serving concept concerned with time to first byte during AI inference.
Inference performanceTime to first token
Time to first token, or TTFT, is the elapsed time from an inference request being accepted until the first output token is delivered. It captures startup and queue delay before generation becomes visible to a user.
Inference performanceTime to last token
Time to last token is the serving concept concerned with time to last token during AI inference.
Inference performanceTimeout budget
Timeout budget is the serving concept concerned with timeout budget during AI inference.
Inference performanceToken streaming
Token streaming is the serving concept concerned with token streaming during AI inference.
Inference performanceTokens per second
Tokens per second is the serving concept concerned with tokens per second during AI inference.
Inference performanceTool latency
Tool latency is the time an application or agent waits for a tool call to complete. It includes work such as a database query, API request, file operation, or test run between model responses.
Inference performance