A dependency-free Python evaluation aid accompanying How to measure whether AI code review is doing useful work. It summarizes evidence you supply. It does not call a model, contact Weave, upload data, estimate employee savings, or detect bugs.
The included data is synthetic. Fictional Tool A and Tool B are examples, not disguised real vendors. No customer outcomes are represented.
Use Python 3.10 or later from this directory:
python3 analyze_reviews.py example-ledger.json --output example-results.json python3 -m unittest discover -s . -p 'test_*.py'
Files: analyzer, example input, example output, behavior checks.
Follow the example JSON structure. Use timezone-aware ISO timestamps. The cohort includes non-excluded PRs merged at or after the period start and before its exclusive end. Pre-merge feedback on those PRs forms the review population; it may have started before the period. Cost must be explicitly matched to that same population before ratios are interpreted.
| Record | Required information |
|---|---|
| Dataset | Label, synthetic boolean, currency, period, pull_requests, reviews, comments, costs. Keep synthetic:true for examples; use false only for an actual observed ledger. |
| PR | Unique id, author_id, merged_at (null if unmerged), optional excluded flag. Period is based on merge time. |
| Review | Unique id, pr_id, reviewer_id, reviewer_kind (human/ai), group, estimated_expert_review_hours (number or null), and assessment_source for a supplied number. Use one record per PR/reviewer, with all rounds consolidated. Estimates must share the same defined assessment method for meaningful comparisons. |
| Comment | Unique id, review_id, kind (suggestion/question/praise/discussion), created_at. Suggestions also need resolution (code_change/reply/unaddressed/unknown) and issue_id (or null). Addressed suggestions require evidence_ref and evidence_at, strictly after the comment and before merge. |
| Issue identity | An auditor’s classification, scoped to a PR. Reuse the same identifier for equivalent findings from different reviewers. Multiple issues cannot be represented by one comment row; split a compound suggestion into separately audited units and document that decision. |
| Cost | Unique id, group, cost_pool_id, net nonnegative amount, currency, scope (review_only/mixed_use), fidelity (billed/estimated/configured), evidence_ref, selected boolean, matched_to_review_scope boolean. Exactly one selected record per pool. Already-netted credits are included in amount; a negative total requires a separate accounting treatment. Reconcile periods, identities and overlapping sources before setting the matching flag. |
Addressed suggestions include both code and reply evidence. The output preserves those categories. Unknowns remain in the all-suggestion denominator and have a separate coverage measure. Questions and praise do not inflate suggestion counts.
Unique implemented issues count only code-addressed suggestions with an issue identifier. Counts associated with individual tools can overlap; use unique_implemented_issues_across_groups for the total. The script does not assign exclusive causal credit.
Cost per estimated expert-review hour requires positive selected cost, review-only scope, declared matching coverage and an assessment for every included review record in that group. Cost per implemented issue has an independent denominator: it requires positive selected cost, review-only scope, declared matching coverage, complete issue identifiers and assessed resolution. It does not require an expert-effort assessment. A blank ratio means the conditions are not satisfied or its denominator is zero, not zero cost or zero value. Human labor cost is not inferred.
The script checks accounting conditions, not the truth of the underlying judgments or the completeness of the source export. Inspect the evidence. Keep customer identifiers, private source code and review content private unless separately cleared for release.
Weave metric definitions: Code Review Output, comment resolution, review quality, AI report. This kit’s unique-issue accounting is an evaluation convention, not a claim that Weave exposes an identically named product metric.