Find a term
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
Terms beginning with Q
3 termsQuantization
Quantization represents model values with lower numerical precision than the original model. It can reduce memory requirements and sometimes improve serving efficiency, with possible effects on output quality and supported operations.
Inference performanceQuantization aware serving
Quantization aware serving is the serving concept concerned with quantization aware serving during AI inference.
Inference performanceQueue latency
Queue latency is the serving concept concerned with queue latency during AI inference.
Inference performance