Cost per completed task

Put unsuccessful attempts in the numerator

The basic calculation is total measured cost divided by successful completions. State which costs are included and how a completion is judged.

For an illustrative benchmark, suppose 20 attempted tasks cost $30 in total and 15 pass the required checks. Cost per completed task is $2. Dividing only the costs of successful attempts by 15 would omit money spent on failures and make the workflow look cheaper than it was.

If no tasks complete successfully, the ratio has no finite value. Report the cost and zero completions directly rather than displaying zero cost per success.

Keep the completion rule stable

A code patch that compiles is not necessarily a completed bug fix. The relevant tests may still fail, or the patch may violate a requirement the grader does not inspect.

Choose the completion criteria before comparing systems. Use the same workload, attempt allowance, and grading process. Otherwise, an apparent cost advantage may reflect an easier definition of success.

Decide what the boundary includes

A narrow benchmark might include only provider charges. A production analysis might also include sandbox execution, retrieval, and human verification. Both can be useful if the labels make the boundary clear.

Avoid mixing units. A task can contain many turns, and a turn can contain several calls. Lower cost per token or per request does not guarantee lower cost per completion when the number of calls and the failure rate change.