Agent Task Outcome

Outcome is different from activity

A coding agent may inspect files, edit code, and announce that a bug is fixed. Those actions provide evidence about its process. The outcome depends on what the patch actually does and whether it meets the requested conditions.

Anthropic's evaluation guide separates an agent's interaction record from the final state produced by the attempt. That distinction prevents a convincing completion message from being treated as proof. For coding work, inspect the diff, execute relevant checks, and verify any required state change independently.

The outcome also depends on the task boundary. Producing a reviewable patch, merging a pull request, and deploying a working feature are different objectives. A successful patch-generation task does not imply a successful production deployment.

Define acceptance before execution

A practical outcome record should identify the task, attempted configuration, resulting artifact, checks performed, and evaluator decision. Use categories that reflect the actual workflow, such as accepted, partially completed, failed, blocked by the environment, and not evaluated. Keep the reason alongside the category.

Separate mandatory conditions from diagnostic observations. For example, preserving an API contract may be mandatory, while the number of tool calls is merely descriptive. An unavailable test service should produce an explicit evaluation gap, not an automatic assertion that the code is correct.

SWE-bench uses repository issues and generated patches as its evaluation setting. A team can apply the same idea to its own tasks without assuming that benchmark success establishes readiness for its production environment.

A hypothetical outcome calculation

Suppose an agent receives 12 tasks to correct date handling in an application. Before execution, the team requires each patch to pass the reported regression case, preserve existing date behavior, and satisfy review.

Eight patches meet all three conditions. Two fail regression tests, one introduces an incompatible API change, and one cannot be checked because its test environment fails to start.

The report should show eight accepted outcomes from 12 submitted tasks, including one unresolved evaluation. It can additionally report eight successes among 11 evaluated tasks, but must label that denominator. Silently removing the blocked attempt makes operational reliability harder to assess.

If an engineer later repairs a failed patch, preserve both records: the original agent outcome and the eventual assisted completion. Combining them into a single success conceals the human contribution.

Limits of an outcome label

A pass records compliance with the checks that were actually applied. It cannot guarantee that no defect exists. Review may miss a problem, tests may omit an edge case, and requirements may change after evaluation.

Keep later reverts or production defects associated with the original work when evidence supports that connection. Report the observation window so a recently accepted change is not treated as having the same production history as an older one. This makes outcomes useful for learning which tasks an agent handles reliably without overstating what has been verified.