Software Factory Feedback: Turn Review Evidence Into PR Standards
Turn recurring review problems into repository standards engineers and coding agents can apply.
On this page
When the same review problem keeps appearing, turn it into a repository standard that engineers and coding agents can apply on the next pull request. Weave helps teams identify recurring PR and review problems, investigate them in Wooly and review later changes against a written standard with Weave Checks.
Engineering Intelligence helps find the problem in the work. Wooly helps investigate it with source-linked answers. Weave Checks reviews changes against standards the team writes in Markdown. Use the PRs and review comments to define the requirement, write it as a Check and evaluate the findings on later changes.
Checks leaves comments; an engineer or coding agent makes the fix. Existing tests, branch policies and deployment approvals continue to determine what reaches production.
Start with a recurring problem in one repository
Choose a problem specific enough that the team can recognize another occurrence. “Improve code quality” is too broad. “Our database migrations repeatedly need another review because they break compatibility during rolling deployments” gives the review a clear subject.
Use Weave to examine the work behind that concern. Compare the same repository, team and completed time window. A four-week baseline is one possible starting point, not a required duration or a guarantee of enough evidence. Record the work mix, tool coverage and release conditions with the numbers. Keep the review-timing mode and weekend settings consistent across periods.
Three useful signals answer different questions:
Scroll horizontally if needed →
| Signal | What Weave measures | What to inspect |
|---|---|---|
| Review turnaround | Time to an eligible reviewer's first review or PR comment after a request; code-update mode measures each response from the preceding push | Initial response waiting; inspect later discussions separately for recurring concerns |
| Code review output | Estimated expert effort represented by the review comments, reviewed diff and correctness reasoning | Whether reviewers repeatedly spend effort explaining the same repository convention |
| Bug ratio | Bug-fix Code Output divided by total merged, nonignored Code Output | Whether more completed work is being consumed by repairs |
Open the relevant PRs before choosing an intervention. A queue can grow because ownership is unclear, changes are too large, a release is blocked or the implementation repeatedly violates a standard. Each needs a different action.
Inspect the PRs and review comments behind the problem
Weave's Code Output estimates expert engineering effort, giving the team a way to weigh the work behind repeated reviews. Inspect the score explanation and diff alongside the review comments. The intelligence-layer guide explains the output unit and the wider cost, quality and delivery evidence.
Ask Wooly a narrow question such as: “Which changes in this repository needed repeated review, and what concerns appeared in those reviews?” Follow the source links into the PRs and discussions. The question is an example of a team investigation, not an automated diagnosis of every delay.
For the migration example, look for concrete compatibility problems: a removed column still used by an older application version, a tightened constraint before a backfill, or a test that never exercises existing data. Record the evidence that connects the repeated review to the proposed standard.
The next action may be a smaller PR, a reviewer assignment, a training session or a repository rule. Choose the action the evidence supports. A new check is useful when it captures a recurring, reviewable requirement.
How GitHub changed its code-review instructions
GitHub's July 2026 code-review account describes a tool migration that initially raised cost and reduced useful findings in offline benchmarks. Traces exposed broad repository browsing; the team changed instructions to gather more focused PR evidence. GitHub then reported roughly 20% lower average production review cost versus its control and said it found no quality signal that would block shipping. That result belongs to GitHub's particular workflow, with no public sample count or test duration in the account.
Inspect the captured coding-agent conversation to see whether repeated broad searches, missing context or unclear instructions contributed to the problem. For coding agents, Weave's captured sessions expose the conversation and decisions where capture is enabled. Its AI report puts model cost per session beside first-shot success, AI output per session and AI code turnover. Use those sessions and the resulting PRs to decide whether to change the instructions or add a repository standard.
Write a focused Markdown standard for Weave Checks
Checks stores one standard per Markdown file in .weave-checks. The pinned Checks documentation requires name, description and intelligence in the rule's frontmatter. Keep the rule focused enough that another engineer can explain why a finding is correct.
This illustrative file could be named .weave-checks/migration-compatibility.md:
--- name: Migration Compatibility description: Flags production schema changes that break existing callers during rollout intelligence: medium ---When to check
Changes to persisted production schemas or migration code.
What to look for
- Identify dropped or renamed fields and tightened constraints.
- Check whether existing callers can still read and write during rollout.
- Flag a concrete incompatibility when no staged migration or maintenance plan addresses it.
- Check for tests covering existing data and the changed behavior.
What not to flag
- Temporary schemas and test fixtures.
- An explicitly planned breaking migration with a documented maintenance window.
How to report
Point to the changed code and explain the incompatibility. State which compatibility evidence or test is missing. Avoid a finding based only on a migration filename or an unfamiliar pattern.
This is a review standard, not a database-safety guarantee. Database tests and deployment checks still need to exercise the actual migration. Have the repository's engineers review the rule before adopting it.
The example leaves the optional files field unset so the reviewer can inspect repository context. A scoped files glob limits the supplied diff and disables repository-reading and shell tools. When no changed files match, Checks skips the rule. Use that option when the check can make its judgment from the matching changes alone.
Check the rule against real examples
Select previously reviewed changes that the rule should flag and changes it should allow. Include a legitimate exception. Check whether the rule flags the intended compatibility problems and allows valid changes before running it on every new PR.
For each example, record the expected result and whether Checks flagged it, allowed it, skipped it, failed with an error or returned an unusable result. Keep the run link or artifact and the applicable rule revision. A skipped rule has not tested the example; an error or unusable result does not show that the change meets the standard.
For each finding, check whether it identifies a real compatibility problem, points to useful evidence and suggests a change the author can act on. Keep rejected findings and the reason for rejecting them. Repeated vague warnings are a reason to revise the standard.
Checks can run locally or through the repository's GitHub Actions workflow. Use the existing setup and CLI documentation rather than copying a second review system into the repository. Model usage and any GitHub Actions minutes have a cost, so observe the cost of representative diffs before widening the rule.
Keep findings and merge policy explicit
Checks posts COMMENT reviews and can resolve its threads after the relevant changes are addressed. It does not rewrite the implementation. Its fail-on-findings option defaults to false; required checks and branch protection are separate repository decisions.
The team can make a standard visible on every PR while retaining a clear owner for exceptions and merge decisions. An automatically resolved comment also does not replace the test result or prove the production deployment succeeded.
Record who owns the rule, what evidence justifies an exception and when the team will revisit it. Keep that information alongside the rule so a future engineer can understand the decision.
Check whether the recurring review concern became less frequent
Return to the same repository and work scope after the team has used the rule. Inspect whether that particular concern appears less often, whether the findings remain useful and whether authors still need repeated explanations.
Read review turnaround alongside the actual conversations. A shorter queue can reflect different work, reviewer availability or a changed release schedule. Treat a before-and-after difference as evidence to investigate, rather than automatically assigning the whole change to the new rule.
Keep later quality signals on their own clocks. Weave's code turnover follows meaningful changes that are subsequently changed or deleted within 90 days. A newly merged change has not completed that follow-up window. Likewise, merge-to-deployment timing requires connected deployment data and matching.
If the concern improves but review noise rises, revise the check. If the same issue persists, inspect whether the rule lacks context or the underlying constraint is elsewhere. Decide whether to keep, revise or remove the Check, or address reviewer assignment, PR size or training instead.
How Intellistack and Remi used Weave findings
Intellistack used Weave to connect AI-tool usage with code output by component. Backend teams were benefiting more than frontend teams, giving leadership a concrete training priority. The public case also describes removing hours of monthly preparation for board reporting. It does not report a fixed productivity percentage or an autonomous factory conversion.
Remi combined GitHub-linked output trends with short weekly Notion reports. Managers combined output trends with each engineer's weekly account before starting a coaching conversation.
The migration standard above is an illustrative workflow, not a claim that either customer runs that check.
For the broader pilot and rollout sequence, use the existing AI-assisted development rollout guide. To evaluate a review tool itself, use the review-impact article and its practical kit.
Download the working review template to record the evidence, rule revision, Checks configuration, expected and observed findings, and follow-up decision.
Start with one repository, a recurring review concern and a few relevant PRs. Use Weave to inspect output, review and delivery evidence; write the standard the team actually wants; then review its findings and the subsequent work before expanding it.
See the work behind your AI spend
Compare code output, AI cost and quality, then inspect the work behind the trend.