Inference performance glossary
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
All terms
100 termsMixed precision inference
Mixed precision inference is the serving concept concerned with mixed precision inference during AI inference.
Inference performanceModel loading
Model loading is the serving concept concerned with model loading during AI inference.
Inference performanceModel parallelism
Model parallelism is the serving concept concerned with model parallelism during AI inference.
Inference performanceModel replica
Model replica is the serving concept concerned with model replica during AI inference.
Inference performanceModel replication
Model replication is the serving concept concerned with model replication during AI inference.
Inference performanceModel warmup
Model warmup is the serving concept concerned with model warmup during AI inference.
Inference performanceOutput length
Output length is the serving concept concerned with output length during AI inference.
Inference performanceP50 latency
P50 latency is the serving concept concerned with p50 latency during AI inference.
Inference performanceP95 latency
P95 latency is the serving concept concerned with p95 latency during AI inference.
Inference performanceP99 latency
P99 latency is the serving concept concerned with p99 latency during AI inference.
Inference performancePaged attention
Paged attention is the serving concept concerned with paged attention during AI inference.
Inference performancePipeline parallelism
Pipeline parallelism is the serving concept concerned with pipeline parallelism during AI inference.
Inference performancePrefill decode disaggregation
Prefill decode disaggregation is the serving concept concerned with prefill decode disaggregation during AI inference.
Inference performancePrefill phase
Prefill phase is the serving concept concerned with prefill phase during AI inference.
Inference performancePrefix cache
Prefix cache is the serving concept concerned with prefix cache during AI inference.
Inference performanceProcessing time
Processing time is the serving concept concerned with processing time during AI inference.
Inference performancePrompt caching
Prompt caching reuses processing associated with previously supplied prompt content, often a matching prefix, to reduce repeated input work. It differs from response caching, which returns a stored answer instead of generating a new one.
Inference performancePrompt length
Prompt length is the serving concept concerned with prompt length during AI inference.
Inference performanceQuantization
Quantization represents model values with lower numerical precision than the original model. It can reduce memory requirements and sometimes improve serving efficiency, with possible effects on output quality and supported operations.
Inference performanceQuantization aware serving
Quantization aware serving is the serving concept concerned with quantization aware serving during AI inference.
Inference performanceQueue latency
Queue latency is the serving concept concerned with queue latency during AI inference.
Inference performanceRequest coalescing
Request coalescing is the serving concept concerned with request coalescing during AI inference.
Inference performanceRequest rate
Request rate is the serving concept concerned with request rate during AI inference.
Inference performanceRequest scheduler
Request scheduler is the serving concept concerned with request scheduler during AI inference.
Inference performanceResponse cache
Response cache is the serving concept concerned with response cache during AI inference.
Inference performanceRetry amplification
Retry amplification is the serving concept concerned with retry amplification during AI inference.
Inference performanceScale to zero
Scale to zero is the serving concept concerned with scale to zero during AI inference.
Inference performanceSemantic caching
Semantic caching stores a response and retrieves it for a later request judged similar in meaning. It uses a similarity method rather than requiring the later request to match the earlier text exactly.
Inference performanceSequence packing
Sequence packing is the serving concept concerned with sequence packing during AI inference.
Inference performanceServer sent events
Server sent events is the serving concept concerned with server sent events during AI inference.
Inference performanceService time
Service time is the serving concept concerned with service time during AI inference.
Inference performanceServing backpressure
Serving backpressure is the serving concept concerned with serving backpressure during AI inference.
Inference performanceSpeculative decoding
Speculative decoding is the serving concept concerned with speculative decoding during AI inference.
Inference performanceStatic batching
Static batching is the serving concept concerned with static batching during AI inference.
Inference performanceStreaming inference
Streaming inference is the serving concept concerned with streaming inference during AI inference.
Inference performanceTail latency
Tail latency is the serving concept concerned with tail latency during AI inference.
Inference performanceTensor fusion
Tensor fusion is the serving concept concerned with tensor fusion during AI inference.
Inference performanceTensor parallelism
Tensor parallelism is the serving concept concerned with tensor parallelism during AI inference.
Inference performanceThroughput latency tradeoff
Throughput latency tradeoff is the serving concept concerned with throughput latency tradeoff during AI inference.
Inference performanceThroughput per gpu
Throughput per gpu is the serving concept concerned with throughput per gpu during AI inference.
Inference performanceTime to first byte
Time to first byte is the serving concept concerned with time to first byte during AI inference.
Inference performanceTime to first token
Time to first token, or TTFT, is the elapsed time from an inference request being accepted until the first output token is delivered. It captures startup and queue delay before generation becomes visible to a user.
Inference performanceTime to last token
Time to last token is the serving concept concerned with time to last token during AI inference.
Inference performanceTimeout budget
Timeout budget is the serving concept concerned with timeout budget during AI inference.
Inference performanceToken streaming
Token streaming is the serving concept concerned with token streaming during AI inference.
Inference performanceTokens per second
Tokens per second is the serving concept concerned with tokens per second during AI inference.
Inference performanceTool latency
Tool latency is the time an application or agent waits for a tool call to complete. It includes work such as a database query, API request, file operation, or test run between model responses.
Inference performanceWarm start
Warm start is the serving concept concerned with warm start during AI inference.
Inference performanceWebsocket streaming
Websocket streaming is the serving concept concerned with websocket streaming during AI inference.
Inference performanceWeight only quantization
Weight only quantization is the serving concept concerned with weight only quantization during AI inference.
Inference performance