Semantic caching
Also known as Semantic response caching, LLM semantic cache
Definition
Semantic caching stores a response and retrieves it for a later request judged similar in meaning. It uses a similarity method rather than requiring the later request to match the earlier text exactly.
Similar meaning is not identical context
An exact response cache usually requires a matching key or request. A semantic cache compares a new request with stored requests or representations and returns a prior response when the similarity is above a configured threshold.
That shortcut works best for stable questions with answers that do not depend on hidden state. It is risky when a request includes a changing account, repository, permission, date, or policy. Two prompts can be similar in wording while requiring different answers.
Set boundaries around reuse
Include relevant context in the cache key or metadata. Set an expiration policy and define how updates invalidate old responses. A threshold that is too low may miss savings. One that is too high may return an answer that sounds plausible but does not solve the current task.
Log whether a response came from cache, the similarity decision, the cache version, and the eventual outcome. Sample misses and hits for review so the hit rate does not become the only success measure.
Do not confuse two kinds of caching
Prompt caching reuses processing for repeated input content while still generating a new answer. Semantic caching can return a previous answer. They can be used together, but they solve different problems and should be measured separately.
How this relates to Weave
Semantic caching can lower repeated request cost, but a cached answer is only useful when it is correct for the current context. Weave Router evaluations should separate cache hits from fresh generations and check task outcomes for both paths.
Explore Router