A reference from Weave

Find a term

Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.

Search results

100 terms

  • Accelerator utilization

    Accelerator utilization is the serving concept concerned with accelerator utilization during AI inference.

    Inference performance

  • Activation quantization

    Activation quantization is the serving concept concerned with activation quantization during AI inference.

    Inference performance

  • Adaptive batching

    Adaptive batching is the serving concept concerned with adaptive batching during AI inference.

    Inference performance

  • Arithmetic intensity

    Arithmetic intensity is the serving concept concerned with arithmetic intensity during AI inference.

    Inference performance

  • Async inference

    Async inference is the serving concept concerned with async inference during AI inference.

    Inference performance

  • Attention optimization

    Attention optimization is the serving concept concerned with attention optimization during AI inference.

    Inference performance

  • Autoscaling

    Autoscaling is the serving concept concerned with autoscaling during AI inference.

    Inference performance

  • Batch inference

    Batch inference processes multiple model inputs together in one serving operation. Grouping requests can improve hardware utilization, but it may add waiting time while a batch fills and must account for different input and output lengths.

    Inference performance

  • Batch padding

    Batch padding is the serving concept concerned with batch padding during AI inference.

    Inference performance

  • BF16 inference

    BF16 inference is the serving concept concerned with bf16 inference during AI inference.

    Inference performance

  • Cache eviction

    Cache eviction is the serving concept concerned with cache eviction during AI inference.

    Inference performance

  • Cache hit rate

    Cache hit rate is the serving concept concerned with cache hit rate during AI inference.

    Inference performance

  • Cache invalidation

    Cache invalidation is the serving concept concerned with cache invalidation during AI inference.

    Inference performance

  • Cache key

    Cache key is the serving concept concerned with cache key during AI inference.

    Inference performance

  • Cache ttl

    Cache ttl is the serving concept concerned with cache ttl during AI inference.

    Inference performance

  • Cache warming

    Cache warming is the serving concept concerned with cache warming during AI inference.

    Inference performance

  • Cold start

    Cold start is the serving concept concerned with cold start during AI inference.

    Inference performance

  • Compute bound inference

    Compute bound inference is the serving concept concerned with compute bound inference during AI inference.

    Inference performance

  • Continuous batching

    Continuous batching is the serving concept concerned with continuous batching during AI inference.

    Inference performance

  • Data parallelism

    Data parallelism is the serving concept concerned with data parallelism during AI inference.

    Inference performance

  • Decode phase

    Decode phase is the serving concept concerned with decode phase during AI inference.

    Inference performance

  • Disaggregated serving

    Disaggregated serving is the serving concept concerned with disaggregated serving during AI inference.

    Inference performance

  • Dynamic batching

    Dynamic batching is the serving concept concerned with dynamic batching during AI inference.

    Inference performance

  • End to end inference latency

    End to end inference latency is the serving concept concerned with end to end inference latency during AI inference.

    Inference performance

  • Expert parallelism

    Expert parallelism is the serving concept concerned with expert parallelism during AI inference.

    Inference performance

  • Fair queuing

    Fair queuing is the serving concept concerned with fair queuing during AI inference.

    Inference performance

  • Flash attention

    Flash attention is the serving concept concerned with flash attention during AI inference.

    Inference performance

  • FP16 inference

    FP16 inference is the serving concept concerned with fp16 inference during AI inference.

    Inference performance

  • GPU memory utilization

    GPU memory utilization is the serving concept concerned with gpu memory utilization during AI inference.

    Inference performance

  • Inference admission

    Inference admission is the serving concept concerned with inference admission during AI inference.

    Inference performance

  • Inference concurrency

    Inference concurrency is the serving concept concerned with inference concurrency during AI inference.

    Inference performance

  • Inference load shedding

    Inference load shedding is the serving concept concerned with inference load shedding during AI inference.

    Inference performance

  • Inference priority queue

    Inference priority queue is the serving concept concerned with inference priority queue during AI inference.

    Inference performance

  • Inference queue time

    Inference queue time is the serving concept concerned with inference queue time during AI inference.

    Inference performance

  • Inference throughput

    Inference throughput is the amount of model inference work completed in a period of time. It may be expressed as requests per second, input tokens per second, output tokens per second, or another workload-specific measure.

    Inference performance

  • Inference token budget

    Inference token budget is the serving concept concerned with inference token budget during AI inference.

    Inference performance

  • INT4 inference

    INT4 inference is the serving concept concerned with int4 inference during AI inference.

    Inference performance

  • INT8 inference

    INT8 inference is the serving concept concerned with int8 inference during AI inference.

    Inference performance

  • INTer token latency

    INTer token latency is the serving concept concerned with inter token latency during AI inference.

    Inference performance

  • Kernel fusion

    Kernel fusion is the serving concept concerned with kernel fusion during AI inference.

    Inference performance

  • KV cache

    KV cache is the serving concept concerned with kv cache during AI inference.

    Inference performance

  • Latency breakdown

    Latency breakdown is the serving concept concerned with latency breakdown during AI inference.

    Inference performance

  • Latency budget

    Latency budget is the serving concept concerned with latency budget during AI inference.

    Inference performance

  • LLM latency

    LLM latency is the time associated with receiving a language model response. Common measures include time to first token, time between generated tokens, and time to the final token, each describing a different user experience.

    Inference performance

  • Long context window

    Long context window is the serving concept concerned with long context window during AI inference.

    Inference performance

  • Max concurrent requests

    Max concurrent requests is the serving concept concerned with max concurrent requests during AI inference.

    Inference performance

  • Max tokens

    Max tokens is the serving concept concerned with max tokens during AI inference.

    Inference performance

  • Memory bandwidth

    Memory bandwidth is the serving concept concerned with memory bandwidth during AI inference.

    Inference performance

  • Memory bound inference

    Memory bound inference is the serving concept concerned with memory bound inference during AI inference.

    Inference performance

  • Microbatching

    Microbatching is the serving concept concerned with microbatching during AI inference.

    Inference performance