LLM latency

Speed has several clocks

Time to first token measures how long a caller waits before output begins. Completion latency measures when the full response arrives. Inter-token timing describes the pace of streaming output between those points.

These measures matter differently by workflow. A chat interface may feel responsive when the first token arrives quickly. A coding agent that must parse a complete tool call may care more about the time to a valid completion.

Include the request and the route

An illustrative comparison sends short prompts to two models and concludes that one is faster. A production agent may send long context, wait in a provider queue, execute tools, and make several calls. The result can change substantially.

Record input size, output size, cache state, selected model, provider, retries, and tool time. Report percentiles because a mean can hide occasional delays that users notice.

Optimize the whole task

Lower latency is not automatically better if the response fails the task or causes another attempt. Compare time to a useful outcome, not just time to a partial stream. Routing can reduce latency for routine work while reserving slower, more capable models for difficult steps.