A reference from Weave

Find a term

Explore latency, throughput, caching, and serving techniques. Understand which parts of a model request consume time and compute.

Terms beginning with C

9 terms

  • Cache eviction

    Cache eviction is the serving concept concerned with cache eviction during AI inference.

    Inference performance

  • Cache hit rate

    Cache hit rate is the serving concept concerned with cache hit rate during AI inference.

    Inference performance

  • Cache invalidation

    Cache invalidation is the serving concept concerned with cache invalidation during AI inference.

    Inference performance

  • Cache key

    Cache key is the serving concept concerned with cache key during AI inference.

    Inference performance

  • Cache ttl

    Cache ttl is the serving concept concerned with cache ttl during AI inference.

    Inference performance

  • Cache warming

    Cache warming is the serving concept concerned with cache warming during AI inference.

    Inference performance

  • Cold start

    Cold start is the serving concept concerned with cold start during AI inference.

    Inference performance

  • Compute bound inference

    Compute bound inference is the serving concept concerned with compute bound inference during AI inference.

    Inference performance

  • Continuous batching

    Continuous batching is the serving concept concerned with continuous batching during AI inference.

    Inference performance