How to measure whether AI code review is doing useful work
Measure AI code-review effort, addressed suggestions and cost without double-counting findings. Includes a runnable synthetic review-ledger example.
On this page
A team paying for several AI code reviewers needs to know which ones improve the code it ships. Comment counts give a poor answer. A reviewer can leave a dozen minor suggestions while another catches the defect that would have reached production. Two tools can also flag the same problem, leaving one fix counted twice.
Weave helps engineering teams evaluate those contributions. It assesses the expert effort represented by human and AI reviews, follows what authors did with suggestions, and brings recorded review-tool spend into the same analysis. Those connections help a team decide which reviewer to keep, where to change its configuration and which work still needs closer human attention.
Examine the work behind each review
Weave’s Code Review Output estimates the expert-engineer effort represented by a reviewer’s submissions. The assessment considers the code under review, the correctness reasoning and the feedback. Human and AI reviewers use the same effort unit.
A bare approval contributes approximately zero. A short PR with difficult security implications can receive a substantial review-effort estimate. Engineers can inspect the reasoning, and administrators can correct an estimate that misses the work involved. The documented measure excludes self-reviews, draft PRs and ignored PRs.
Open the reviews behind a tool’s output and check whether they contain the work you value: identifying an unsafe assumption, explaining a failure case or suggesting a practical correction.
The unit is estimated expert effort. Two expert-review hours describe the assessed substance of the review; they are not two observed hours of employee time saved. Review waiting time belongs in a separate delivery measure.
Follow suggestions through to a decision
An author can implement a suggestion, explain why the current code is correct, defer the change with a reason or leave it unanswered. Each outcome tells you something different about the review.
Weave’s comment resolution rate looks for later commits or clear replies. Clicking “resolved” alone does not establish that a suggestion was addressed. The measure covers eligible suggestions on merged PRs; questions, praise and general discussion are excluded. Group AI comments by tool to examine each reviewer’s results.
For an evaluation, record implemented changes separately from replies. A well-reasoned explanation can resolve a concern without changing the code. Calling every such outcome an “AI suggestion shipped” would overstate what happened. Keep a link to the relevant commit or discussion so another engineer can check the judgment.
Missing evidence should remain visible. If a suggestion has no assessed outcome, record it as unknown and retain it in the population. Otherwise the percentage can rise simply because difficult cases were dropped.
Count a shared finding once
When two tools identify the same missing authorization check, the author makes one fix. Both reviewers may have contributed, but the combined result is one implemented issue.
A simple review ledger can capture this with an issue identifier for each distinct problem on a PR. Comments about the same problem share that identifier. Inspect the code and discussion before grouping them; similar wording alone is not enough.
Consider a synthetic example. Tool A leaves six suggestions. Three lead to code changes, but two of those comments concern the same problem, so they represent two implemented issues. Tool B leaves four suggestions, three of which lead to three implemented issues. One issue appears in both tools’ findings. Together, the tools are associated with four distinct implemented issues.
Tool A’s remaining suggestions have one clear reply, one unaddressed outcome and one unknown outcome. Four of its six suggestions were addressed through code or discussion, and five of six outcomes were assessed. Reporting both counts shows the unresolved evidence instead of hiding it inside a percentage.
Human feedback can overlap with these findings too. Preserve that association. Tool-level totals help explain each reviewer’s contribution, while the combined total needs the duplicate issues removed. The ledger cannot establish which reviewer caused a fix.
Compare the cost of the same work
Weave’s AI report has a dedicated Code review tab for tools such as CodeRabbit, Greptile and Cursor Bugbot, alongside review activity from broader coding assistants. Its views connect recorded spend, review output, addressed comments and review cycles. Review-only tools are separated from tools that also author code.
For your evaluation, match costs to the reviewed population. If a provider invoice and a usage estimate describe the same expense, choose one as the cost source and use the other to check it. Adding both would double the bill. A subscription that covers coding and review also needs an explicit allocation before you treat it as review-only spending.
In the synthetic example, Tool A costs $120 for 1.2 estimated expert-review hours. Tool B costs $180 for 2.4. Their costs are $100 and $75 per estimated expert-review hour. Tool B produces more assessed review work per dollar in this example, although that alone says nothing about the severity of its findings.
Calculate that ratio only when the selected costs cover the same review work and every included review has an effort assessment. If coverage is incomplete, show the cost, output and missing assessments separately until the gap is resolved.
Judge the findings that matter
A security defect and a naming suggestion should not carry equal weight in a buying decision. Weave’s review-quality assessment considers depth, communication, practicality and demonstrated impact. It excludes self-reviews, approval-only reviews and reviews of PRs with Code Output below one expert-effort hour. Its eligible review population therefore differs from Code Review Output.
Read the high-value findings and the disagreements between reviewers. Check whether the advice was correct, whether the proposed change was practical and whether it addressed a problem worth fixing. A tool that repeatedly contributes useful, distinct findings deserves a different decision from one that repeats low-value feedback already supplied elsewhere.
Reviewing the comments that exist cannot reveal every defect a tool missed. To measure missed-bug recall, use a separate set of changes with independently established defects. Keep that test alongside the review evidence when deciding how much responsibility to give an automated reviewer.
Connect review work with delivery delays
A reviewer can leave useful feedback while the PR still waits for someone to act. Weave’s review-turnaround measure tracks the median time until an eligible reviewer responds with a review or PR comment. The timer starts at the review request, or at the latest code update when that mode is selected. Compare the delay with the feedback itself before changing the reviewer, assigning ownership or adding review capacity.
At OpenZeppelin, Daniel Carmo used Weave to investigate a project problem and estimated that catching it saved about four weeks of engineering time. The published case shows why earlier investigation matters. That estimate concerns one project; the synthetic reviewer comparison above illustrates the separate cost-accounting method.
Use the evaluation to change something
Start with merged PRs from a defined period and include the kinds of changes your team normally ships. Record one review per reviewer per PR, with its pre-merge feedback, effort assessment, suggestion outcomes and cost coverage. Exclude self-review and document any omitted repositories or files.
Review the results with the engineers who handled those PRs. Use the findings to make a specific change: disable a noisy rule, retain a tool that catches distinct problems, adjust a review policy or add human review to an area with weak coverage. Then evaluate the next comparable period using the same definitions.
Weave gives that discussion a useful starting point: inspectable review work, the author’s response and the cost of the tools involved. Ask to see those connections on your own PRs when you evaluate Weave Engineering Intelligence.
Run the review-ledger example to reproduce the synthetic accounting above. The kit summarizes supplied assessments; it does not reproduce Weave’s assessment models.
See the work behind your AI spend
Compare code output, AI cost and quality, then inspect the work behind the trend.