Tool latency

The model is not the whole clock

An agent's visible response time includes more than generation. The model may issue a tool call, wait for a shell command, receive a large result, and then reason about the next step. Tool latency is the time consumed by that outside operation.

For example, a coding agent can produce a tool call quickly but still take thirty seconds to run a test suite. Measuring only time to first token would describe the beginning of the turn while missing the wait that determines when useful work is finished.

Measure each tool separately

Record the tool name, request type, start and finish timestamps, success status, and result size. Percentiles often tell a clearer story than an average because a few slow API requests can make an agent feel unreliable.

Tracing connects tool spans to the model turns that caused them. That context helps distinguish a slow provider from a slow repository operation and shows whether retries are adding more tool work.

Optimize the workflow

Parallel calls can reduce elapsed time when the tools are independent. Smaller queries and targeted test commands can reduce work when they preserve the needed signal. Any optimization should still be checked for correctness, because skipping a validation step can create a faster but less useful task.