Find a term
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
Terms beginning with P
11 termsP50 latency
P50 latency is the serving concept concerned with p50 latency during AI inference.
Inference performanceP95 latency
P95 latency is the serving concept concerned with p95 latency during AI inference.
Inference performanceP99 latency
P99 latency is the serving concept concerned with p99 latency during AI inference.
Inference performancePaged attention
Paged attention is the serving concept concerned with paged attention during AI inference.
Inference performancePipeline parallelism
Pipeline parallelism is the serving concept concerned with pipeline parallelism during AI inference.
Inference performancePrefill decode disaggregation
Prefill decode disaggregation is the serving concept concerned with prefill decode disaggregation during AI inference.
Inference performancePrefill phase
Prefill phase is the serving concept concerned with prefill phase during AI inference.
Inference performancePrefix cache
Prefix cache is the serving concept concerned with prefix cache during AI inference.
Inference performanceProcessing time
Processing time is the serving concept concerned with processing time during AI inference.
Inference performancePrompt caching
Prompt caching reuses processing associated with previously supplied prompt content, often a matching prefix, to reduce repeated input work. It differs from response caching, which returns a stored answer instead of generating a new one.
Inference performancePrompt length
Prompt length is the serving concept concerned with prompt length during AI inference.
Inference performance