Agent Developer Productivity

Measure the developer and agent together

An agent can generate a patch quickly while creating hours of investigation for its reviewer. Another agent may take longer to run but leave a small, well-tested change that is easy to accept. A useful productivity measure includes both experiences.

Choose a unit of completed work before measuring improvement. For a controlled exercise, that might be a specified bug fix that passes tests and review. For ongoing team reporting, use a stable output definition and separate features, maintenance, and incidents. Comparing raw pull request counts can be misleading when teams split changes differently.

Human effort includes task preparation, supervision, review, and corrections. Keep elapsed delivery time separate: an unattended agent run or a queue for review can extend delivery time without consuming an equivalent amount of developer attention. Tool spending is another input and belongs in an accompanying cost calculation.

An illustrative calculation

Suppose a team evaluates two matched sets of ten maintenance tasks. These numbers are hypothetical, not benchmark results.

• Without agents, ten accepted tasks require 40 human hours.
• With agents, ten accepted tasks require 24 hours of implementation and supervision, plus 8 hours of review and correction.
• Output per human hour changes from 10 / 40 = 0.25 to 10 / 32 = 0.3125 accepted tasks.

That is a 25% increase in output per hour and a 20% reduction in hours for the same output. The percentages answer different questions. Reporting only the 24 implementation hours would overstate the improvement by omitting review and correction.

Check the tasks again after a defined observation period. If additional defects require four hours of repairs, include those hours in the relevant cohort and report the revised result.

Establish a credible comparison

Record task type, repository, developer experience, acceptance criteria, and which agent configuration was available. Prefer comparable work and, where practical, randomized assignment over a simple comparison of enthusiastic adopters with non-users. Preserve unsuccessful attempts instead of measuring only merged successes.

GitHub's metrics documentation distinguishes adoption and engagement from acceptance and pull request activity. Those categories provide useful observations, but a high usage count does not itself establish an increase in completed value.

Where estimates can mislead

METR's 2026 experiment update describes selection effects when developers decline participation or withhold tasks they prefer to perform with AI. It also identifies difficulties measuring human time when developers operate several agents concurrently. These are practical reasons to document participation and time accounting, not reasons to substitute opinions for measurement.

Treat improvements as specific to the tested people, tasks, tools, and period. Productivity evidence should guide workflow decisions and investigation. It should not turn prompt volume or an isolated output score into an individual performance ranking.