Agent traffic is 62% cache reads, and it should change how you think about routing

There's a mental model of LLM routing that goes like this: a request arrives, you score it, you send it to the best model for the job. Every request is a fresh decision. It's a clean picture, and for agent traffic it's wrong.

Here's the number that breaks it. Over the last 30 days, of the roughly 46 billion input-side tokens that flowed through the Weave router, 28.8 billion were prompt-cache reads. That's 62.1% of everything models ingested. Only 17.6 billion tokens were fresh input, and another 5.3 billion were cache writes.

In other words, most of what a coding agent sends to a model is something the model has already seen. The agent's context grows turn by turn: the system prompt, the conversation so far, the tool results piling up. Providers charge a tenth or less for reading cached tokens versus fresh ones, and agent harnesses lean on that heavily. A typical multi-turn session pays full price for its context once and then re-reads it at a steep discount for every subsequent turn.

The catch

Prompt caches don't transfer between models. The cache lives with the provider, keyed to the exact prefix of the conversation. The moment you switch a session from model A to model B, model B has seen none of it. Your next request pays full freight on the entire accumulated context.

So the clean per-request routing picture has a hidden cost term. Mid-session, "the best model for this request" is usually the model you're already on, because switching means re-buying a few hundred thousand tokens of context at the fresh-input rate. The router that naively picks the cheapest capable model on every turn will torch its own savings on cache misses.

What we do instead: session pins

The Weave router treats the session, not the request, as the unit of routing. The first turn of a session gets a genuine routing decision. Then the session is pinned, and subsequent turns ride the pin unless something breaks it.

The production numbers over the last 30 days:

• 14,316 sessions were pinned, averaging 10.4 turns per pin

• 33.6% of all requests were served straight off a sticky pin, no fresh decision needed

• Only 1.4% of pinned sessions ever switched models mid-session

That last number is the one we watch. It could be low for a bad reason (the router refusing to correct mistakes) or a good one (first-turn decisions being good enough that corrections are rarely worth the cache penalty). Two things point to the second explanation. Failover, where we switch because the upstream provider is erroring or overloaded, fired on just 0.28% of requests in the period, so pins almost never break for reliability reasons. And the router keeps making shadow decisions on pinned turns, scoring what it would have picked fresh; the pinned model and the fresh choice agree far more often than not.

Switching is still sometimes right

A pin is a default, not a cage. There are moments where eating the cache penalty is the correct trade. If a session shows signs of struggle, repeated errors, thrashing on the same file, the value of a stronger model outweighs re-reading context. If a provider melts down, obviously you move. And when a session escalates from quick exploration into deep multi-file work, the task has genuinely changed shape.

The point isn't that you should never switch. It's that switching has a price tag, and that price tag is invisible if your routing math treats every request as independent. At 62% cache reads, the independence assumption isn't a simplification, it's a tenfold billing error on the input side.

The takeaway for anyone building a router

If you're routing chat traffic, per-request routing is fine. Sessions are short, contexts are small, caches barely matter.

If you're routing agent traffic, the cache is the terrain. Any routing policy that doesn't model it will look great in a spreadsheet of per-request prices and lose money in production. Route the first turn hard, pin the session, and make un-pinning a deliberate decision with the cache penalty priced in.

We got to 62% cache reads without trying, just by serving real coding agents. Whatever that fraction is on your traffic, it's probably higher than your routing math assumes.