Find a term
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
Terms beginning with A
7 termsAccelerator utilization
Accelerator utilization is the serving concept concerned with accelerator utilization during AI inference.
Inference performanceActivation quantization
Activation quantization is the serving concept concerned with activation quantization during AI inference.
Inference performanceAdaptive batching
Adaptive batching is the serving concept concerned with adaptive batching during AI inference.
Inference performanceArithmetic intensity
Arithmetic intensity is the serving concept concerned with arithmetic intensity during AI inference.
Inference performanceAsync inference
Async inference is the serving concept concerned with async inference during AI inference.
Inference performanceAttention optimization
Attention optimization is the serving concept concerned with attention optimization during AI inference.
Inference performanceAutoscaling
Autoscaling is the serving concept concerned with autoscaling during AI inference.
Inference performance