AI-assisted development: how to roll out and measure impact

AI-assisted development means using AI to help with software work, including understanding code, implementing changes, writing tests, and reviewing pull requests. To roll it out across a team, start with a specific workflow, establish a baseline, and judge the result across cost, completed work, quality, and delivery. More usage alone does not establish that the investment worked.

TL;DR

• Start with a recurring engineering problem and a team that can evaluate the result.
• Define what success looks like before the pilot. Include review effort and reliability alongside speed.
• Keep interactive assistance and autonomous agent work separate when their supervision, cost, and risk differ.
• Compare similar work over consistent reporting windows. Treat a before-and-after change as an observation, then investigate other explanations.
• Expand a workflow when the evidence supports it. Fix the bottleneck or change the approach when it does not.

What changes when a team adopts AI-assisted development?

AI assistance can happen throughout the development process. An engineer might ask for an explanation of an unfamiliar service, draft a regression test, or work through a small refactor. An autonomous agent might make a change and submit a pull request for human review.

These workflows need different evaluation criteria. An explanation should help an engineer reach a correct understanding. A test should catch the failure it was written for. An agent-authored change should meet the same acceptance criteria and release checks as other production work.

The unit of adoption is the workflow you are trying to improve. Buying seats across the organization does not establish which workflow improved or where the benefits stopped.

DORA's 2025 State of AI-assisted Software Development report describes AI as amplifying an organization's existing strengths and weaknesses. Its practical implication for a rollout is to examine the surrounding system. Faster implementation can still leave work waiting for a reviewer, a test environment, or a release.

Step 1: choose the decision and the pilot

Write the decision the pilot should answer. For example: “Should this team expand AI assistance for regression tests?” That is more useful than “Should everyone use more AI?”

Choose a recurring workflow with a clear completion condition. Good candidates have enough context for an engineer to check the result and enough comparable work to observe a pattern. Possible starting points include test maintenance, repository navigation, documentation tied to a code change, and bounded bug fixes. These are candidates to evaluate, not a promise that every assistant handles them well.

Record a short pilot brief:

• Problem: the current source of delay, effort, or errors.
• Scope: the team, service, task types, and approved tool configuration.
• Acceptance: what a completed task must do and who checks it.
• Comparison: how you will compare the work with the existing workflow.
• Constraints: data access, permissions, spending, and release requirements.
• Decision owner: the person who will expand, revise, or stop the pilot.

Give the pilot enough time for work to complete and for follow-up problems to appear. A short demo can establish that a tool works on a task. It cannot establish the effect on a team's delivery system.

Step 2: make the workflow usable and reviewable

Before measuring adoption, remove avoidable ambiguity. Engineers should know which tools they can use, what data they can provide, and which actions require a human decision.

Put the relevant repository instructions, build commands, test commands, and acceptance criteria where the assistant can use them. Maintain that context alongside the code. Broken setup instructions create friction for engineers and agents alike.

For an autonomous workflow, specify its boundary. Define which repositories it can access, whether it can write changes, and who can approve a release. Preserve the evidence a reviewer needs: the task, the diff, the checks run, and any limitations.

These recommendations align with the areas examined in DORA's AI Capabilities Model, including clear AI guidance, usable internal information, and development infrastructure. The checklist above is a starting point for your own rollout, rather than a reproduction of DORA's model.

Train with a real task from the pilot. Walk through the context, an incorrect suggestion, the correction, and the final review. A useful session teaches engineers how to recognize a result they should reject.

Step 3: establish a measurement baseline

Use the same definitions before and during the pilot. Record the service, reporting dates, eligible tasks, team membership, and tool configuration. Keep missing data visible.

Use a small set of measures that answer different questions:

DORA's current delivery guidance uses five measures: change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate. Apply them to the service being changed. DORA cautions against comparisons across unlike applications and recommends interpreting several measures together.

You do not need to collect every possible metric. For a test-writing pilot, start with accepted tests, the behavior they cover, engineer and reviewer effort, and spending. Add delivery and reliability measures where the pilot can plausibly affect them.

Step 4: reconcile spending and attribution

Keep the financial record separate from the activity record. An invoice answers what the organization paid. A usage export answers what a vendor recorded. Connecting the two may require a mapping from accounts to people and teams.

Jellyfish's guide to AI coding tool costs describes a useful sequence: inventory the charges, attribute spending to teams, and compare it with delivery data. Use your actual contracts and invoices for the amounts. Vendor price lists and token estimates are different cost bases.

For each cost source, record the reporting window, currency, included allowance, extra usage, and shared charges. Explain how you allocate a pooled account. If you cannot allocate it reliably, retain it as a shared cost rather than inventing individual precision.

Apply the same care to AI attribution. A native tool event, a commit co-author marker, and an inferred activity window carry different evidence. Missing attribution should remain unknown. It should not silently become “human only.”

Separate interactive assistance from autonomous agents when they require different amounts of supervision. Include the engineering time spent setting up, reviewing, correcting, and maintaining the workflow when deciding whether it is worthwhile.

Step 5: investigate the result before expanding

Compare the pilot with its baseline using the same inclusion rules. Check whether task difficulty, staffing, release practices, or incident load changed. Separate the learning period from later use if onboarding materially affected the result.

A staggered rollout or comparable team can help you investigate other explanations. Neither makes the groups identical. A simple before-and-after chart can show an observed change; it does not isolate AI as the cause.

Read representative changes with the engineers and reviewers. Look for where a local improvement became extra work elsewhere:

Agree on decision rules before reviewing results. Expand when the workflow meets its acceptance criteria at an acceptable cost and quality level. Revise it when the bottleneck is identifiable. Stop or limit it when the evidence does not support further investment.

Step 6: make rollout a repeatable operating practice

Keep a short decision record for each expansion: what changed, what evidence supported it, who owns it, and when to review it. Carry forward the test, review, and release requirements that made the pilot trustworthy.

Revisit a workflow after a material tool, model, pricing, or repository change. A result from one configuration is not a permanent property of “AI.” Share examples of useful work and rejected output so teams can learn without repeating the same failures.

The goal is a better engineering workflow. Some tasks may remain better suited to the existing process. Let the evidence support that decision too.

Where Weave fits in an AI-assisted development rollout

Token Intelligence connects AI adoption and spending with engineering output. It distinguishes direct tool integrations, commit metadata, and estimated attribution. Cost-source labels separate provider billing, modeled usage, and estimates so you can inspect the basis of a report.

Code Intelligence adds a common output measure. Silk 1 reads code changes to estimate the expert engineering effort represented by the work, with explanations that teams can inspect. That is a modeled measure of work, not a measurement of hours saved or cash returned.

Use those views alongside Engineering Intelligence to examine reviews, quality, and delivery. The AI ROI methodology explains how cost and output fit together. For a vendor evaluation, the AI ROI measurement tools guide compares the documented approaches.

Weave provides evidence for the rollout decision. Your team still needs to define acceptance, interpret the work, and verify whether the change helped users.

Book a demo to review how your team's AI usage, costs, and engineering output can be evaluated together.

Frequently asked questions

How should a company start AI-assisted development?

Choose a recurring problem, a bounded workflow, and a team that can check the result. Record the existing performance, define completion and quality requirements, and compare the pilot before expanding it.

How do you measure the impact of AI-assisted development?

Connect adoption and spending to accepted work, review effort, delivery, reliability, and the relevant user outcome. Keep definitions and observation windows consistent. Investigate other changes before attributing an observed improvement to AI.

Is more AI-generated code evidence of higher productivity?

It is evidence of generated volume. To assess productivity, examine whether useful work is completed with acceptable cost, review effort, and quality. Generated work that must be corrected or discarded still consumes resources.

Should autonomous agents and coding assistants use the same scorecard?

They can share outcome and quality measures. Track their usage, spending, supervision, and attribution separately when the workflows differ, then compare completed work on a consistent basis.