Find a term
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
Terms beginning with B
3 termsBatch inference
Batch inference processes multiple model inputs together in one serving operation. Grouping requests can improve hardware utilization, but it may add waiting time while a batch fills and must account for different input and output lengths.
Inference performanceBatch padding
Batch padding is the serving concept concerned with batch padding during AI inference.
Inference performanceBF16 inference
BF16 inference is the serving concept concerned with bf16 inference during AI inference.
Inference performance