Find a term
Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.
Terms beginning with S
9 termsScale to zero
Scale to zero is the serving concept concerned with scale to zero during AI inference.
Inference performanceSemantic caching
Semantic caching stores a response and retrieves it for a later request judged similar in meaning. It uses a similarity method rather than requiring the later request to match the earlier text exactly.
Inference performanceSequence packing
Sequence packing is the serving concept concerned with sequence packing during AI inference.
Inference performanceServer sent events
Server sent events is the serving concept concerned with server sent events during AI inference.
Inference performanceService time
Service time is the serving concept concerned with service time during AI inference.
Inference performanceServing backpressure
Serving backpressure is the serving concept concerned with serving backpressure during AI inference.
Inference performanceSpeculative decoding
Speculative decoding is the serving concept concerned with speculative decoding during AI inference.
Inference performanceStatic batching
Static batching is the serving concept concerned with static batching during AI inference.
Inference performanceStreaming inference
Streaming inference is the serving concept concerned with streaming inference during AI inference.
Inference performance