LLM Cost Optimization and Model Routing: A Practical Guide for Engineering Leaders

What is AI token cost?

AI token cost is the amount a model provider charges for processing input and generating output. The bill depends on the model, token counts, caching, and applicable service tier. A request can also incur tool or storage charges. Check the provider's current OpenAI, Anthropic, or Gemini pricing before calculating a budget.

For engineering teams, the useful question is how much it costs to complete work at the required quality. A lower token bill is valuable when the code still passes its checks and the team can use it.

To optimize LLM cost, measure the full cost of a task first. Then test one change: reduce unnecessary context, reuse eligible prompt prefixes, batch work that can wait, or route suitable tasks to another model. Compare accepted results, retries, and latency alongside the bill.

Where Weave fits

Weave connects available AI coding-tool usage and spend with engineering output; attribution coverage depends on the connected tool. Token Intelligence provides the cost and usage view, while Engineering Intelligence adds review, quality, and delivery context. Weave Router routes coding-agent requests between models and reports routing decisions and costs. Platform plans and Router subscriptions are billed separately; see current pricing for the product you are evaluating.

For one published example, Weave Router solved 62.1% of 66 Terminal-Bench 4.0 tasks within two attempts at $5.22 per trial, compared with GPT-6 Astra's 60.6% at $10.03. The runs used the same Codex harness and paired tasks. See the full benchmark and methodology; results on a published task set are a starting point, so evaluate representative work before changing a routing policy.

Build a cost baseline you can explain

Record the model, billable token categories, pricing date, owner, task, retries, and result. Keep coding subscriptions, metered API charges, and other infrastructure costs identifiable so the same expense is counted once.

For each token category, multiply its billable count by the applicable rate. If the rate is quoted per million tokens, divide the count by one million first. Add the resulting charges and any applicable non-token fees.

Provider billing categories differ:

Do not charge the same input tokens twice. For example, OpenAI's caching documentation distinguishes uncached-input, cached-input, and cache-write rates rather than adding a cache-write fee to the base rate. Reconcile your calculation with billing before using it to compare tools.

How to calculate AI token cost

For a text workload with these billing categories, use:

Token cost = (uncached input tokens × input rate + cache-write tokens × write rate + cache-read tokens × read rate + output tokens × output rate) ÷ 1,000,000.

The rates in this formula are prices per million tokens. Split the token counts into the provider's mutually exclusive billing categories first. If your usage total already includes cached input, subtract the cached categories before applying the ordinary input rate. Add tool calls, storage, and other applicable charges separately.

The following is an illustrative arithmetic example, with invented rates and token counts. It is not a provider quote, customer result, or Weave benchmark.

This example assumes no tool, storage, or other fees. A single average token price would hide the difference between output, cache reads, and ordinary input. Use actual billed categories and the applicable model, context length, service tier, and pricing date when reproducing the calculation.

Measure cost per accepted task

Cost per accepted task = cost of all attempts in the workload ÷ number of accepted tasks.

For example, if the hypothetical $9.05 workload above includes every attempt and produces eight accepted tasks, its token cost per accepted task is $9.05 ÷ 8, or about $1.13. Include spending on failed attempts in the numerator. Report human review time, infrastructure, and subscriptions separately unless you have included those costs in the calculation.

Choose the control that fits the problem

Cost monitoring, model routing, and engineering measurement answer different questions. Use this checklist when evaluating a tool or building your own workflow.

Compare the ways to reduce LLM cost

Choose the change that addresses the part of the bill you measured. The table below compares mechanisms and the evidence to collect; it does not promise a savings rate for your workload.

Provider billing and feature documentation was reviewed on October 16, 2026. Rates, eligibility, context thresholds, and processing modes can change. Check the linked source for the model and API you actually use.

Evaluate routing and caching together

Model routing selects a model for a request. Prompt caching reuses eligible input context. Compare their combined cost on representative tasks: switching models can change cache reuse, while failed attempts can add requests and review work.

Start with a small evaluation set from the work your team actually does. Define success before testing a cheaper model: passing tests, accepted changes, required latency, or another task-specific result. Compare total cost across all attempts needed to reach that result. Include unsuccessful runs in the spending total.

Keep an escalation path for tasks that miss the threshold. Re-run the evaluation when models, prompts, pricing, or workload mix change. A published benchmark can inform a shortlist; your own acceptance results determine whether a routing policy fits your work.

Connect spending to engineering output

Choose one cohort and time window, then review its costs, delivered work, and quality together. Keep unattributed spending visible. Recently merged code may need more time before defects or maintenance work appear.

Cost per merged PR can be useful for a consistent workload, but PR size and complexity vary. Review the underlying work alongside the ratio. Weave's AI ROI methodology explains its output estimates and the distinction between observed cost efficiency and a financial return calculation.

Common questions

Will a cheaper model always reduce the cost of a task?

Compare the full run. Extra attempts, cache changes, and human corrections can offset a lower rate. Measure cost per completed task alongside quality and latency.

Can token counts prove AI ROI?

Token counts describe consumption. A financial return calculation also needs attributable costs, a credible baseline, and a value for incremental benefits. Review Weave's measurement methodology before interpreting an output ratio as ROI.

What should we bring to a demo?

Bring the coding tools you use, team size, billing sources, and one decision you need to make, such as a renewal or routing rollout. Book a Weave demo to review the integrations and measurement approach for your team.