Code Generation Rework

Separate correction from normal development

Generated code often changes before it ships. Some edits correct mistakes, while others adapt an exploratory draft into a finished design. The distinction matters when a team claims that an agent saves time or creates additional work.

Define the intended deliverable first. If an agent is asked for a prototype, replacing its temporary implementation during planned production work is not necessarily rework. If it is asked for a production-ready patch and omits a required access check, fixing that omission is corrective work.

Code churn is broader than rework. Formatting, file moves, refactoring, and new requirements can all change recently written code without indicating a generation failure. Classify the reason for an edit rather than using a time-based deletion count as proof of a defect.

Choose a measurable unit

Two useful measures answer different questions:

• Affected-change rate: generated changes requiring correction divided by eligible generated changes reviewed during the period.
• Correction effort: human time spent diagnosing and fixing generation problems, with any separately measured agent cost reported alongside it.

Keep the observation window explicit. A pre-merge measure captures review corrections but misses production defects. A 30-day follow-up provides more history, while still missing later discoveries. Compare cohorts with equivalent observation periods.

Google's review guidance offers categories such as functionality, design, and testing that can help organize correction reasons. A review comment itself is not a unit of rework: several comments may describe one issue, and some merely ask for clarification.

A hypothetical rework calculation

A team reviews 40 AI-assisted changes during a pilot. Eight need corrections to satisfy their original acceptance criteria. The affected-change rate is 8 / 40, or 20%.

Engineers spend ten hours diagnosing and repairing those eight changes. Separately, they spend six hours adding newly requested functionality to three other changes. The pilot reports ten hours of correction effort, not sixteen, because the additional functionality changed the requirements.

One corrected change requires seven hours; the remaining seven need three hours combined. Reporting only the 20% rate would hide this concentration. Inspect the expensive failure to determine whether a task specification, tool constraint, or review check could prevent it.

These figures are illustrative. They do not establish that using agents increased total effort: a comparable workflow without agents may also require corrections.

Account for attribution limits

Keep links between original suggestions, accepted patches, reviews, and corrective changes where the workflow supports them. When humans and several agents edit the same code, provenance may not identify which contribution introduced a problem. Report an AI-assisted change with corrective work rather than assigning unsupported blame to a specific model.

METR's experiment update highlights the difficulty of recording human time when developers run multiple agents concurrently. Avoid counting all elapsed agent time as human repair effort. Combine the correction measure with accepted output and total development effort to assess whether the workflow improved overall productivity.