Code Generation Quality

Quality depends on the intended change

A generated function can compile and still implement the wrong behavior. A patch can pass a regression test while adding an unnecessary dependency or bypassing a repository's authorization pattern. Evaluation should therefore begin with the requested behavior and the constraints of the receiving system.

Define which conditions are mandatory for this change. A payment calculation may require explicit rounding rules, while an internal command-line tool may need clear failure messages and safe handling of local files. Applying the same generic score to both can hide the properties that matter most.

Combine evidence from several checks

Automated tests provide evidence about behavior under the conditions they exercise. Type checking and static analysis expose other classes of problems. Human review examines whether the implementation fits the surrounding code and the intended design.

Google's code review guidance includes design, functionality, complexity, testing, naming, and documentation. That breadth is useful for generated changes because syntactically plausible code can still be difficult to maintain. Review should assess the patch with the same substantive standards used for other contributions.

Keep the underlying observations available if a dashboard combines them into a score. A weighted average can conceal a critical failure: strong readability does not compensate for an authorization bypass. Treat required security or correctness checks as gates rather than interchangeable points.

A worked hypothetical review

Suppose an agent generates a cache for customer records. It passes 18 existing tests and reduces repeated database queries in a local performance check. Those results initially look promising.

The reviewer notices that the cache key contains the record identifier but omits the tenant identifier. Two tenants using the same record identifier could receive each other's data. A new isolation test fails, so the patch is rejected despite passing the original suite.

After the key is corrected, the isolation test passes. The reviewer then checks expiration behavior and documentation before accepting the change. The outcome is a revised patch that satisfies the stated checks, not a claim that the original 18-test pass represented complete quality.

This example is illustrative. It shows why a quality report should identify the test scope, failures found, and corrections made instead of publishing only a pass percentage.

Observe what happens after acceptance

Repository benchmarks such as SWE-bench evaluate generated patches against defined issues. They provide bounded evidence about task performance, not a complete assessment of every production constraint.

After merge, inspect relevant reverts, defects, and corrective edits over a stated period. Distinguish a defect repair from an intentional redesign or changing requirement. A high change rate is a reason to investigate, not automatic evidence of poor generation.

Finally, assess quality at the patch and workflow level. Several models and human edits may contribute to one change. Attribute a problem to a particular generator only when the available provenance supports that conclusion, and keep uncertainty visible when it does not.