Lines of code, prompts and accepted suggestions measure activity, not progress. Here is what the research says about AI coding tools and a four-layer scorecard to use instead.

To measure engineering productivity with AI coding tools, track delivery speed, stability and product outcomes, not lines of code, prompts or accepted suggestions. Those are activity signals, and AI inflates them for free. Controlled research shows why it matters: in METR's 2025 trial, experienced developers were 19% slower with AI while believing they were 20% faster. Measure the system around the tool, in four layers.

Key takeaways

  • Perceived and measured productivity can point in opposite directions: METR's developers forecast a 24% speedup, estimated 20% afterwards, and measured 19% slower (METR, 2025).
  • On high-AI teams, merged pull requests rose 98% while review time rose 91% and PR size 154%, with no company-level delivery gain (Faros AI, 2025).
  • Google's own guidance says acceptance rate and accepted lines of code measure adoption and trust, not productivity.
  • DORA's 2025 research links AI adoption to higher throughput but lower stability, so measure both sides.
  • Use a four-layer scorecard: adoption, trust, delivery, product impact. Give a pilot six to eight weeks against a baseline.

Why are lines of code a bad measure for AI-assisted engineers?

Lines of code were always a weak proxy. With AI coding tools they become actively misleading, because generating code now costs almost nothing. A dashboard can fill up with commits, pull requests, generated tests and accepted suggestions while the product stands still.

Three pieces of evidence show the distortion.

  • More code is not better code. GitClear analysed 211 million changed lines from 2020 to 2024. Cloned (copy-pasted) lines rose from 8.3% of changes in 2021 to 12.3% in 2024, while moved, refactored lines fell from 25% to under 10% (GitClear, 2025). Volume went up as maintenance work went down.
  • Bigger changes slow the team. Faros AI's telemetry from more than 10,000 developers found high-AI teams produced PRs 154% larger, review time rose 91% and bugs per developer rose 9% (Faros AI, 2025).
  • Acceptance is not productivity. Google's framework for adopting AI coding assistance places acceptance rate and lines of code accepted in its adoption and trust phases, and warns that high values do not prove productivity if delivery metrics worsen (Google Cloud, 2025).

The SPACE framework, from Nicole Forsgren and colleagues at Microsoft Research and GitHub, made the general point in 2021: developer productivity cannot be measured by a single metric or by activity levels alone (Forsgren et al., 2021). Its five dimensions are satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Activity is one of five, not the whole picture.

The risk is a feedback loop: the organisation rewards visible output, engineers generate more, reviewers inherit the load, and leaders mistake motion for progress. We call this the product velocity illusion.

What does the research say about AI coding tools and developer productivity?

The results depend heavily on the task, the codebase and the developer. Here is the evidence side by side.

Study Setting Finding
Peng et al., 2023 Controlled task: build an HTTP server in JavaScript Copilot group finished 55.8% faster (arXiv)
METR, July 2025 RCT: 16 experienced developers, 246 real issues in large open-source repos 19% slower with AI; developers believed 20% faster (METR)
METR, February 2026 57 developers, 800+ tasks, late-2025 tools Estimates of 18% and 4% faster, but confidence intervals include zero; METR calls it weak evidence (METR)
DORA, 2024 Survey-based modelling AI adoption associated with an estimated 1.5% drop in delivery throughput and 7.2% drop in stability (Google Cloud)
DORA, 2025 Nearly 5,000 professionals AI now linked to higher throughput, still to lower stability (Google Cloud)
Faros AI, 2025 Telemetry, 10,000+ developers, 1,255 teams 21% more tasks, 98% more PRs, no company-level gain
Stack Overflow, 2025 Survey, 30,000+ developers 52% say AI improved their productivity; 45% say debugging AI code takes longer (Stack Overflow)

Three lessons for managers:

  1. Self-reports are not enough. METR's perception gap means a team survey alone can show gains that delivery data does not.
  2. Isolated tasks overstate gains. A greenfield exercise with clear requirements is where AI shines. Mature codebases with implicit conventions are harder.
  3. Individual gains do not add up automatically. Faros found more output per developer but no improvement at company level. Review, testing and integration absorb the difference.

DORA's 2024 data also showed real benefits: for every 25% increase in AI adoption, documentation quality rose an estimated 7.5%, code quality 3.4% and code review speed 3.1%. AI helps. The question is whether your system turns that help into delivered value.

What should you measure instead? A four-layer scorecard

Google's four-phase framework (adoption, trust, acceleration, impact) and DORA's delivery metrics combine into a scorecard you can run monthly. Keep each layer separate so adoption numbers never stand in for impact.

Layer Question Metrics Warning sign
1. Adoption Are engineers using the tools? Weekly active users, share of changes with AI assistance Treated as a target in itself
2. Trust Do engineers trust the output appropriately? Acceptance rate, survey on trust, share of AI changes reworked in review Very high acceptance plus rising change fail rate
3. Delivery Is the system faster and stable? DORA's five metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate (DORA) Throughput up, stability down
4. Product impact Did customers or the business benefit? Target user behaviour, adoption of shipped features, retention, support load Many releases, flat outcomes

Add two team-health signals that AI tends to move: median PR size and median time to first review. They are the earliest warnings that output is outrunning review capacity.

How do you run a fair before-and-after comparison?

Google's guidance names defining success criteria and a baseline as the most important and most often skipped step. A practical sequence:

  1. Agree the success criteria with engineers and business leads before rollout. Pick one delivery metric and one product outcome.
  2. Capture a baseline for at least four weeks: DORA metrics, PR size, review time.
  3. Compare like with like. Google warns against comparing an experienced AI-using team with a less experienced team without AI. Use successive projects or matched cohorts.
  4. Give it six to eight weeks. Google suggests at least four two-week sprints before expecting acceleration to show.
  5. Read all four layers together, then decide what to change: tooling, batch size, review process or context.

Worked example (illustrative)

Say a six-engineer team adopts an AI coding agent. After eight weeks, the dashboard shows lines changed up 140% and merged PRs up 60%. Change lead time fell from 3.0 days to 2.2 days. But change fail rate rose from 8% to 14%, median PR size doubled, and time to first review went from 4 hours to 10.

A lines-of-code dashboard calls this a triumph. The four-layer scorecard says adoption and throughput improved while stability got worse. The fix: cap PR size, add the PM's acceptance check before code review, and require agents to list untested paths. Four weeks later, change fail rate is back to 9% and lead time holds at 2.4 days. That is a real gain.

What should a manager agree with the team before AI-assisted work starts?

Make six things explicit for each piece of work:

  1. The product decision: what are we deciding or testing?
  2. The user evidence: which sources show the problem is real?
  3. The smallest useful slice: what is the smallest change that creates learning or value?
  4. The technical guardrails: what must remain true for security, reliability, privacy and performance?
  5. The review path: who reviews behaviour, code, tests and operational consequences?
  6. The outcome owner: who checks what happened after launch?

Then ask the team to make generated work legible: files and systems changed, assumptions the agent made, tests run, untested paths, rollback plan, and a link back to the product decision. Treat generated code as a proposed change, not an authority. The PM's side of this review is covered in how to review what AI agents build.

How should product managers work with engineers using AI coding tools?

Give engineers the problem, the evidence, the acceptance criteria, the constraints and the intended outcome, then let engineering own the technical design. Agents only know what is written down, so the brief carries more weight than it did. A structured AI coding agent PRD template and the product context an AI coding agent needs both help.

Keep slices small. DORA lists working in small batches among the seven capabilities that amplify AI's benefits, alongside a clear AI stance, AI-accessible internal data and strong version control practices.

A weekly management loop

Replace "how many lines did we produce?" with seven questions:

  • What did we learn from customers or production?
  • Which decisions changed?
  • What work was generated but not yet justified by a decision?
  • Which changes are too large to review comfortably?
  • Where did AI remove friction?
  • Where did it add review or operational load?
  • What should we stop, narrow or roll back?

What are the common mistakes?

Setting adoption targets. "80% of engineers using AI daily" is an input. Teams will hit it without improving anything.

Trusting self-reports alone. METR's perception gap is the warning. Pair surveys with delivery data.

Ignoring stability. Throughput gains that come with more rollbacks and hotfixes are borrowed time.

Comparing unlike teams. A senior team with AI against a junior team without it tells you nothing about AI.

Skipping the policy. Define approved tools, data boundaries, review thresholds and the sensitive actions that need human approval. Our guide to how AI changed product team workload covers the wider operating contract.

Where Specky fits

Specky keeps product and engineering connected to the reason for the work. Its Product Graph links customer signals, research, decisions, PRDs, tickets and outcomes, and its MCP server lets coding agents read that context before they plan. That gives managers a loop beyond code activity: signal, decision, implementation, review, release, outcome. It is not another engineering dashboard; it stops the product reasoning getting lost between the roadmap and the repository.

Frequently asked questions

Should managers track lines of code produced by AI?

No. Lines of code and accepted suggestions are activity measures that AI inflates at no cost. GitClear's data shows copy-pasted code rising and refactoring falling as AI use grew. Track DORA's delivery metrics, review time, change size and the customer outcome each change was meant to move instead.

What is the best metric for AI developer productivity?

No single metric works. Use four layers: adoption (are tools used), trust (is output accepted appropriately), delivery (DORA's throughput and stability metrics) and product impact (did customer behaviour or business results change). Report them separately so adoption never stands in for impact.

Do AI coding tools make developers faster?

It depends. A 2023 controlled task found developers 55.8% faster with Copilot. METR's 2025 trial on mature open-source codebases found experienced developers 19% slower. METR's 2026 follow-up points towards modest speedups but with wide uncertainty. Gains are largest on well-defined, greenfield tasks.

How long should an AI coding tool pilot run?

At least six to eight weeks after a baseline period. Google recommends four two-week sprints before judging acceleration, because engineers need time to learn the tools and early numbers mostly reflect adoption. Agree success criteria with engineers and business leads before the pilot starts.

What should an AI coding tool policy include?

Approved tools, the data each tool may access, review thresholds by risk, actions that always need human approval, how generated changes are tested and audited, and who owns production incidents. DORA calls this a clear and communicated AI stance and lists it first among the capabilities that make AI adoption pay off.

Sources

  1. METR — Measuring the impact of early-2025 AI on experienced open-source developer productivity
  2. METR — Uplift update (February 2026)
  3. Peng et al. — The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (arXiv, 2023)
  4. Google Cloud — Highlights from the 10th DORA report (2024)
  5. Google Cloud — Announcing the 2025 DORA report
  6. DORA — DORA's software delivery metrics
  7. Google Cloud — A framework for adopting Gemini Code Assist and measuring its impact
  8. Faros AI — The AI Productivity Paradox Report 2025
  9. GitClear — AI Copilot Code Quality: 2025 research
  10. Microsoft Research — The SPACE of Developer Productivity (ACM Queue, 2021)
  11. Stack Overflow — 2025 Developer Survey: AI

This story was originally published on 15 October 2026 at https://www.specky.space/blog/manage-engineers-ai-coding-tools-without-lines-of-code.