A report on AI productivity, human capability, software economics, SaaS defensibility, and the future of engineering work.
Between 2024 and 2026, the marginal cost of producing a line of working code fell further and faster than at any previous point in the history of the discipline.
Large language models and the coding agents built on top of them can now generate prototypes, interfaces, tests, scripts, documentation, database schemas, migrations, and integrations at a speed that would have been implausible five years earlier.
A reasonable observer, watching a competent agent produce a functioning application in an afternoon, might conclude that software engineering has been solved, or at least commoditized, and that the profession built around it is entering a period of terminal decline.
The evidence assembled in this report points in a different direction. Across randomised controlled trials, large-scale observational studies of developer behaviour, benchmark evaluations of autonomous agents, labour-market projections, payroll microdata, venture funding data, and public-market valuations, a consistent pattern emerges.
Artificial intelligence has substantially reduced the cost of generating code while leaving largely intact (and in several respects increasing) the value of the human capacity required to decide what should be built, to judge whether the output is correct, and to accept responsibility for the result.
A study of more than one hundred thousand developers found that a 180 percent increase in commits produced only a 30 percent increase in actual software releases and no measurable increase in end-user adoption (NBER).
An analysis of approximately four hundred thousand agent sessions found that the same model succeeded roughly twice as often when operated by an expert as when operated by a novice (Anthropic).
Employment projections show the narrow occupation of computer programmer declining by six percent over a decade while the broader occupation of software developer grows by nearly sixteen percent (Bureau of Labor Statistics).
Taken together, these findings describe a bifurcation rather than a collapse. At the lower end of the profession (the translation of clear instructions into syntax, the assembly of standard components, the production of conventional interfaces) work is being automated, compressed, and commoditized at speed.
At the upper end (architecture, requirements definition, security, reliability engineering, domain modelling, verification, integration, and accountability), the evidence suggests that scarcity is increasing rather than decreasing, because the volume of generated artifacts requiring judgment has risen faster than the supply of people qualified to exercise it.
The same bifurcation is visible in the economics of software businesses, where the median public SaaS company saw its revenue multiple fall from 6.7 to 3.6 in a single year while acquirers simultaneously paid an average of 6.3 times revenue for companies embedded in genuinely difficult workflows (Software Equity Group).
It is visible again in what enterprises did once they had the tools to act on the opposite conclusion: in the year of maximum enthusiasm, with better internal code-generation capability than any cohort in the history of the industry, 76 percent of enterprise artificial-intelligence solutions were purchased rather than built (Menlo Ventures).
This report sets out that argument in five parts.
Part I examines what the productivity evidence actually shows, including the substantial gains AI delivers on bounded tasks and the equally substantial losses it can produce on complex ones. It also prices generation rather than assuming it's free, confronts the objection that AI will simply automate its own verification bottleneck, and sets out the security surface that cheap generation creates.
Part II examines human capability as the binding constraint on AI-assisted engineering, the length of time required to build it, the regulated industries in which a licensed human signature isn't merely valuable but legally required, and the class of systems in which the cost of a defect exceeds the cost of construction by several orders of magnitude.
Part III examines the labour market, including the concentration of hiring at senior levels and the structural risk this creates for the future supply of expertise, and closes by asking what a university degree still protects once AI narrows the performance gap it used to certify, and what open learning makes possible for engineers who never enrol.
Part IV examines the economics of software businesses after code became cheap, including the migration of competitive advantage out of the codebase and into the surrounding system, and then examines where investment capital is actually being allocated inside software once hardware-heavy categories are excluded from the count and the heavy overlap between the artificial-intelligence and software classifications is accounted for.
And Part V translates the analysis into practical guidance for engineering organisations, for founders deciding when a prototype must become a company, for individual practitioners, and for capital allocators, and sets out explicitly the conditions under which the central thesis of this report would be falsified.
What We'll Cover:
How to Read This Report
Every quantitative claim in this document is attributed to a named source and linked at the point of use, and the complete set of statistics is reproduced in Appendix A, grouped by subject, so that readers who wish to interrogate the evidence base directly can do so without reading the argument that surrounds it.
Appendix B discusses the quality and limitations of that evidence, including the studies that are vendor-sponsored, the studies with small samples, the studies whose findings have already been partially superseded by newer tooling, and the studies whose authors explicitly caution against the causal interpretations that have been placed on them in public commentary.
Treat that appendix as an integral part of the argument rather than as a disclaimer appended to it, because a report arguing that judgment matters more than output would be self-refuting if it presented its own evidence without judgment.
The report also contains code. The listings are illustrative rather than executable: they are reduced to the smallest form that carries the point being made, they omit the imports, error handling, and configuration that any real version would require, and none of them should be copied into a system. Their purpose is to make the central distinction of this report concrete at the only altitude where it can be seen directly, which is the altitude of the artefact itself.
If you don't want to write software, you can skip every one of them without losing the argument. Each is preceded by the claim it illustrates and followed by the reading of it.
The report deliberately includes findings that cut against its thesis. A randomised trial of ninety-five professional developers found a 55.8 percent improvement in task completion speed with AI assistance, and a separate study of two hundred and two experienced developers found measurable improvements in readability, reliability, maintainability, and conciseness (Microsoft; The GitHub Blog).
These are real results from well-constructed experiments, and any account of the current moment that omits them is propaganda rather than analysis.
The argument advanced here is not that AI fails to deliver value, which would be plainly false. Instead, it's that the value it delivers is task-dependent, organisation-dependent, and operator-dependent in ways that the prevailing discourse has largely failed to price.
Part I: The Cheapening of Code
1. The Attractive, Incomplete Idea
There's a widely held and increasingly confident belief that software engineering has become easy. The argument is straightforward and has the additional advantage of being partly true.
Because AI can now write code, a small team located anywhere in the world, equipped with a few GPUs, access to frontier models, and sufficient ambition, can reproduce in weeks what previously required an experienced engineering organisation and a year of funded effort.
The barriers that once protected incumbent software companies, in this account, were never really technical. They were merely accumulations of labour that machines can now perform at a fraction of the cost. Remove the labour cost and the barrier disappears with it.
The idea is attractive because it contains a genuine insight, and because it flatters an ambition that many people hold and few previously could act on. It is also incomplete in a way that matters enormously for anyone allocating capital, planning a career, designing an organisation, or deciding what to build.
Artificial intelligence has genuinely transformed software engineering. It produces prototypes, interfaces, tests, scripts, documentation, database schemas, and integrations at extraordinary speed. It helps people learn unfamiliar technologies, explore approaches they wouldn't otherwise have considered, and automate work that was previously both necessary and tedious.
Nothing in this report disputes any of that, and a substantial portion of it is devoted to quantifying exactly how large those gains are on the tasks where they occur.
But making easy tasks easier isn't the same thing as making difficult engineering problems disappear, and the conflation of the two is the central error of the current discourse.
The distinction can be stated precisely: AI has lowered the cost of producing code without eliminating the difficulty of building reliable systems or the human capacity required to judge them. Something that looks like functioning software isn't necessarily dependable software, and the gap between the two categories is where the entire economic value of engineering has always resided.
A generated application may work perfectly in a controlled demonstration and still fail under concurrent load, hostile input, partial outage, schema migration, unusual data, changing requirements, compliance review, adversarial attack, or six months of continued development by people who didn't write it and can't easily read it.
What AI has compressed is the distance between an idea and a demonstration, and that compression is genuine, large, and consequential. What it has not compressed, at least not to a comparable degree, is the distance between a demonstration and a dependable enterprise.
The remainder of this report is concerned with measuring that second distance, understanding why it has proven resistant to automation, and drawing out the implications for the people and companies operating on either side of it.
2. What Software Engineering Actually Consists of
Software engineering has never been primarily about producing syntax, though the visibility of syntax has always made it appear that way to observers outside the discipline. Code is one output of an engineering process, and in mature organisations it's rarely the most expensive one.
The actual work also includes:
discovering what the requirements are (which is frequently harder than satisfying them)
selecting an architecture that will remain viable as those requirements change
designing failure boundaries so that the malfunction of one component doesn't propagate through the system
protecting data whose loss or exposure would be catastrophic
controlling dependencies whose maintainers are strangers
testing abnormal conditions that nobody has yet imagined
integrating with systems that were built under different assumptions by people who have left
monitoring production behaviour that differs from every test environment
managing migrations that can't be rolled back
coordinating teams whose incentives are imperfectly aligned
and maintaining the product for years after the enthusiasm of its initial construction has dissipated.
A model can generate thousands of lines of plausible code in a few minutes, and the plausibility isn't superficial. The code will frequently compile, pass the tests it was given, and behave correctly on the inputs anyone thought to try.
What the model doesn't automatically know is which business requirement actually matters to the organisation as opposed to which one was most clearly articulated in the prompt.
It also doesn't know which trade-off the organisation is prepared to accept and which it will reverse at considerable expense six months later. It has no idea which legacy system can't be interrupted because it settles payments, or which regulatory obligation applies in which jurisdiction and what evidence a regulator will demand. And it hasn't a clue which dependency will become a security liability when its sole maintainer abandons it, which architectural decision will create a bottleneck two years and two orders of magnitude of traffic later, and which apparently minor feature will become the central pillar of the product at scale because a category of customer nobody modeled will build their operations on it.
The distance between those paragraphs is easier to see in an artifact than in an argument. Consider the most ordinary requirement in commercial software, stated the way a product manager would state it: let a user move money from one account to another. A capable model answers it in seconds, and the answer isn't bad.
# The generated answer to "let users transfer funds between accounts". # It compiles, it passes the tests it was given, and it is wrong. def transfer(session, from_id, to_id, amount): source = session.query(Account).get(from_id) target = session.query(Account).get(to_id) if source.balance < amount: raise InsufficientFunds() source.balance -= amount target.balance += amount session.commit() return {"status": "ok"} Nothing in that function is incompetent. It states the business rule correctly, it checks the one precondition anybody would think to check, and it will behave perfectly in every demonstration.
It also has at least six defects that will appear in production and not before.
Two concurrent transfers out of the same account read the same balance and both succeed, overdrawing it.
A client that times out and retries debits the account twice, because nothing records that the first request was already applied.
A negative amount reverses the direction of the transfer and is accepted.
A transfer between accounts in different currencies moves the number without converting it.
Two simultaneous transfers in opposite directions between the same pair of accounts acquire their rows in opposite orders and deadlock.
And there is no record of what happened, which means that when one of the preceding five occurs there is no way to establish what the system did.
# The engineered answer. The business rule is unchanged. Everything added # exists because of a specific way the version above fails in production. def transfer(session, from_id, to_id, amount, idempotency_key): if amount <= 0: raise InvalidAmount(amount) prior = session.query(Transfer).filter_by(key=idempotency_key).one_or_none() if prior is not None: return prior.result # a retry of a request already applied first, second = sorted((from_id, to_id)) # fixed lock order, no deadlock with session.begin(isolation_level="SERIALIZABLE"): a = session.query(Account).with_for_update().get(first) b = session.query(Account).with_for_update().get(second) source, target = (a, b) if a.id == from_id else (b, a) if source.currency != target.currency: raise CurrencyMismatch(source.currency, target.currency) if source.balance < amount: raise InsufficientFunds(from_id, amount, source.balance) source.balance -= amount target.balance += amount session.add(Transfer(key=idempotency_key, src=from_id, dst=to_id, amount=amount, at=utcnow())) return {"status": "ok"} The second version isn't harder to generate than the first, and this is the point rather than an objection to it. Any competent model will produce something close to it when the request says so: serialise the transaction, lock in a fixed order, require an idempotency key, reject non-positive amounts, check the currencies, and write an audit row.
The scarce input isn't the ability to write those lines. It's knowing that each of them has to exist, which is knowledge acquired by having watched the first version fail, and which the person writing the prompt either has or doesn't have.
The code became cheap, but the list did not. The complete version, with the exception hierarchy, the retry envelope, the double-entry ledger, and the nine tests that separate it from the first, is reproduced in Appendix D.1. It runs to roughly three hundred lines against the ten that express the requirement.
This is the mechanism by which producing more code fails to translate into producing more value, and it's not a subtle or contested mechanism. It's simply that code is an intermediate good in a production process whose other inputs are scarce. And increasing the supply of an intermediate good doesn't increase output when a different input is binding.
The remainder of this section examines the evidence that this is precisely what has occurred.
3. The Production Funnel: What Happened When 100,000 Developers Received Agents
The most direct large-scale evidence on this question comes from a 2026 study published by the National Bureau of Economic Research, which examined more than one hundred thousand GitHub developers across successive generations of AI coding tools.
The research design exploits the fact that these tools arrived in distinct waves: first autocomplete, then interactive coding agents operating in a conversational loop, then autonomous agents capable of executing multi-step tasks with limited supervision. And this allowed the authors of the study to observe how developer behaviour changed as capability increased.
The effect on code production was unambiguous and large. Autocomplete increased coding activity by approximately 40 percent. Interactive coding agents increased it by approximately 140 percent. Autonomous agents increased it by approximately 180 percent.
These aren't marginal improvements: a near-tripling of commit volume across a population of 100,000 developers represents one of the largest measured productivity shifts in the history of the software industry. Anyone who doubts that AI does something real to the mechanics of programming should sit with that number before proceeding.
Across more than 100,000 developers, autocomplete increased coding activity by approximately 40 percent, interactive coding agents by approximately 140 percent, and autonomous agents by approximately 180 percent. (National Bureau of Economic Research)
What happened next is the finding that matters. The 180 percent increase in commits translated into only about 50 percent more projects and approximately 30 percent more actual releases. Across four major application marketplaces, the researchers observed a moderate increase in the number of new applications published but no corresponding increase in total usage.
The funnel, in other words, narrowed sharply at every stage after generation, and by the time the additional output reached the point at which economic value is actually realised (a customer using the software), the measured effect had disappeared entirely.
The arithmetic is worth stating explicitly because it's easy to lose in the percentages. If commits rise by 180 percent and releases rise by 30 percent, then the ratio of commits to releases has increased by a factor of roughly two and a sixth: each release now consumes substantially more generated code than it did before.
# The NBER funnel stated as arithmetic rather than as percentages, with # every stage indexed to its own pre-agent baseline of 1.00. stage_multiple = { "commits": 2.80, # +180 percent "projects": 1.50, # +50 percent "releases": 1.30, # +30 percent "usage": 1.00, # no measured change } previous = 1.00 for stage, multiple in stage_multiple.items(): print(f"{stage:>9} {multiple:.2f}x " f"passed through: {multiple / previous:.0%}") previous = multiple # commits 2.80x passed through: 280% # projects 1.50x passed through: 54% # releases 1.30x passed through: 87% # usage 1.00x passed through: 77% # # Generated artefacts consumed per shipped release, against the baseline: # 2.80 / 1.30 = 2.15x The column on the right is the one that carries the argument. Each figure is the proportion of the preceding stage's gain that survived into the next, and the conversion is worst at the first transition after generation. That's the throat of the funnel, and it's where the inputs that weren't expanded by the tools (review capacity, requirement clarity, integration effort, and the attention of people who understand the system) do their binding.
Before that ratio can carry any argumentative weight, one measurement question has to be answered, and this report can't fully answer it. A commit is a unit of developer behaviour, not a unit of code, and the tools under study changed the behaviour directly: an agent working through a task in steps produces a commit per step, where a person working through the same task by hand might have produced one at the end.
Some unrecoverable fraction of the 180 percent is therefore an artifact of the instrument rather than a change in what was built, and the size of that fraction determines how much of the attenuation is real.
A study reporting lines changed, or the distribution of commit sizes before and after adoption, would resolve it. Until one does, the true reading is that the direction of the funnel is well established (the ratio of generated artifacts to shipped releases rose, and rose substantially) while the specific factor of two and a sixth should be treated as an upper bound on the true attenuation rather than a measurement of it.
The final stage of the funnel requires a separate explanation, and conflating it with the others would be an error that this report has no interest in committing.
That new applications appeared in marketplaces without total usage rising isn't evidence about engineering capacity at all. Total marketplace usage is bounded by how many people there are and how much attention and money they have, which is a demand-side constraint that no amount of engineering judgment, review capacity, or architectural discipline would relieve.
It's close to definitional that increasing supply into a demand-constrained market doesn't increase aggregate consumption of that market. The usage finding therefore establishes something real and important: that the binding constraint on software adoption was never the supply of software. But it establishes it about customers rather than about engineers, and it shouldn't be counted as support for the thesis of this report.
The stages that do support that thesis are the earlier ones, where generated code failed to convert into completed projects and completed projects failed to convert into releases.
Within those earlier stages, the bottlenecks appeared after generation rather than during it. Requirements still had to be clarified with people who hadn't fully decided what they wanted. Outputs still had to be reviewed by engineers who could evaluate them, and review capacity didn't increase by 180 percent.
Components still had to be integrated with systems that had their own constraints. Products still had to be tested against conditions nobody had specified, secured against adversaries who hadn't been modelled, deployed into environments that behaved differently from staging, supported by people who had to understand them, and, most stubbornly of all, adopted by customers whose attention, budgets, procurement processes, and switching costs were entirely unaffected by the fact that the software had become cheaper to produce.
The study also estimated a substitution elasticity of only 0.25 between AI and human effort. This is a technical parameter with a plain-language meaning: it indicates that AI and human work remain strongly complementary rather than substitutable, and that a company can't straightforwardly trade one for the other at constant output.
An elasticity of that magnitude implies that reducing human input while increasing AI input degrades production substantially, which is a formal statement of the intuition that the two inputs do different things. AI accelerated one part of the production process, and accelerated it dramatically, but the surrounding human system determined how much of that acceleration reached anyone who would pay for it.
The estimated substitution elasticity between AI and human engineering effort was approximately 0.25, indicating strong complementarity rather than straightforward replacement. (National Bureau of Economic Research)
That parameter also creates an obligation this report must discharge in Part III rather than leave hanging, because at first sight it predicts the opposite of what the labour data show. If AI and human effort are strong complements, then making AI abundant and cheap should raise the marginal product of the human input and therefore raise demand for it.
Part III will document falling entry-level hiring and a fifteen-to-one skew toward senior roles, which looks like the signature of substitution rather than complementarity.
The reconciliation is that the estimated elasticity is an aggregate over a workforce that's not homogeneous in this respect, and that it conceals two opposite relationships: the tasks that define junior work are the bounded, well-specified ones on which AI substitutes most readily, while the planning, verification, and accountability that define senior work are the ones with which it composes.
An aggregate elasticity of 0.25 is consistent with a profession in which one half of the workforce is being complemented and the other is being displaced, and that's the reading this report adopts throughout.
The production funnel that emerges from these figures can be stated compactly, and it remains the single most important quantitative expression of the argument in this report, with the qualifications above attached: substantially more generated code produced fewer completed projects than the code volume would suggest, and those projects produced fewer releases still.
The stages between generation and release are the places where human judgment, human coordination, and human attention are the limiting inputs, and none of those inputs was expanded by the arrival of the tools. What happened after release was governed by customer demand, which is a separate constraint and a separate argument.
4. What Generation Actually Costs
The argument to this point has followed the convention of the wider discussion and treated generation as though it were free.
Measured against the wage of a person writing the same code by hand, that convention was defensible when the tools were autocomplete. It's no longer defensible now that the dominant mode is an agent running for minutes or hours against a large context, and the gap between the two framings has grown large enough to change conclusions rather than merely to qualify them.
A report whose title asserts that code is becoming cheaper owes the reader an account of what it actually costs.
The forecast that has focused industry attention came from Gartner in June 2026, which projected that artificial-intelligence coding token costs will meet or exceed the average software engineer's monthly salary by 2028.
The accompanying analysis is explicit that this is a warning about ungoverned consumption rather than a prediction of inevitability, and the analyst's framing is worth quoting because it states the trade-off precisely:
The same analysis puts the productivity contribution of assistive development at up to 20 percent, which is the other half of the ratio and the reason the forecast matters.
Gartner projects that AI coding token costs will meet or exceed the average software engineer's monthly salary by 2028, against a productivity contribution from assistive development of up to 20 percent. (Gartner)
One qualification must travel with that figure wherever it's repeated, and it's routinely dropped. The forecast is anchored on a global average salary of approximately $2,000 per month, which is not a United States crossover point and was never claimed to be. Coverage of the forecast notes that in India the comparison may already hold for an engineer with four to six years of experience, while prevailing salaries in the United States and Western Europe place the crossing considerably further out (CIO).
Read carelessly, the headline says agents will cost more than engineers. Read correctly, it says the ratio of tool cost to labour cost varies by an order of magnitude across the world's engineering labour markets, and that the economics of AI-assisted development are therefore geographically specific in a way the discourse hasn't absorbed.
Published vendor figures give the current level. Enterprise deployments of one widely used coding agent run at approximately $13 per developer per active day and $150 to $250 per developer per month, with ninety percent of users remaining below $30 on any active day (Anthropic).
Heavily automated workflows, in which agents run continuously rather than being invoked, are reported well above that range. Against a fully loaded engineering salary, these aren't yet large numbers. They're also not the rounding error that the phrase "cheap code" implies, and their trajectory is the more interesting quantity.
That trajectory contains a finding which should unsettle anyone assuming the cost problem solves itself. In April 2026 the vendor more than doubled its own published estimate of what an enterprise developer spends, and the reason given was that a more capable frontier model had become the default (Yahoo Finance). A capability improvement raised the cost per developer rather than lowering it.
The mechanism isn't mysterious and is familiar from every previous efficiency gain in computing: when a more capable model arrives, it's not used to perform yesterday's task more cheaply but to attempt harder and longer-running tasks that were previously out of reach. Cost per token falls, while tokens per task rise faster. Anyone forecasting that improving models will make agentic development cheap should first explain why this observation is not the general case.
The demand side confirms the pattern at market scale. Weekly token processing volume on one large routing platform rose from 0.4 trillion in December 2024 to 27.0 trillion by March 2026, an increase of roughly sixty-eight times in fifteen months. Software-engineering agents specifically consume more than a thousand times the tokens of single-turn reasoning, at an input-to-output ratio exceeding 150 to 1 (arXiv).
The input-heavy ratio is the technical signature of agentic work: the model is repeatedly re-reading a large context in order to take one small action, and it's the reason agentic costs bear so little relation to the length of the output produced.
Weekly token volume on a major routing platform rose from 0.4 trillion in December 2024 to 27.0 trillion by March 2026, approximately 68 times in fifteen months, with software-engineering agents consuming over 1,000 times the tokens of single-turn reasoning at an input-to-output ratio above 150 to 1. (arXiv)
The most useful statistic in this literature, however, is not a level or a growth rate but a ratio, and it reproduces the structure of Section 3 in a different currency.
On a standard software-engineering benchmark, one widely studied agent configuration cost approximately $0.24 per instance attempted and approximately $32.50 per issue actually resolved (arXiv). The gap between those two numbers is a factor of roughly one hundred and thirty-five, and it exists for exactly the reason the commit-to-release gap exists: most attempts don't convert.
A buyer looking at the cost of an attempt is looking at the wrong number in the same way that an executive looking at commit volume is looking at the wrong number. The correction in both cases is to divide by the conversion rate rather than to admire the numerator.
# Two ways of pricing the same agent, from the benchmark cost study above. COST_PER_ATTEMPT = 0.24 # dollars, per instance the agent is given COST_PER_RESOLVED = 32.50 # dollars, per issue actually resolved print(f"implied resolution rate: {COST_PER_ATTEMPT / COST_PER_RESOLVED:.1%}") # implied resolution rate: 0.7% # The number an engineering leader actually needs is neither of the above, # because the expensive input is not the model. It is the person who has to # read what the model produced before it can be merged. def cost_per_delivered_change(attempts, unit_cost, merge_rate, review_minutes, reviewer_rate): machine = attempts * unit_cost merged = attempts * merge_rate human = merged * (review_minutes / 60) * reviewer_rate return (machine + human) / max(merged, 1e-9), machine / (machine + human) total, model_share = cost_per_delivered_change( attempts=1_000, unit_cost=0.24, merge_rate=0.30, review_minutes=25, reviewer_rate=95.0) print(f"${total:,.2f} per merged change; {model_share:.1%} of it is inference") # $40.38 per merged change; 2.0% of it is inference The proportions in that calculation are the substance of it rather than the precision. Under assumptions that are unremarkable for an organisation reviewing agent output seriously, the inference bill is two percent of the cost of a delivered change and the review is ninety-eight.
An organisation that halves its token spend has moved one percent of its engineering cost. An organisation that halves the time a competent reviewer needs to reach a confident decision has moved something close to half of it, and that's a tooling, architecture, and testing problem rather than a procurement one.
This has a methodological consequence that the evaluation literature has already drawn. Accuracy measurements that don't control for cost aren't decision-useful, because an agent can be made to look better simply by being allowed to spend more.
When the comparison is made properly, simple baseline configurations turn out to dominate elaborate state-of-the-art agents on some coding benchmarks at approximately fifty times lower cost (arXiv). The recommendation that follows (report the accuracy-cost frontier rather than a leaderboard position) is the same discipline this report argues for on the delivery side in Section 44, arrived at independently by researchers looking at a different problem.
On a coding benchmark, simple baseline agent configurations Pareto-dominated complex state-of-the-art agents at approximately 50 times lower cost, indicating that accuracy figures uncontrolled for cost are not decision-useful. (arXiv)
There's also now a benchmark denominated in the market price of the human labour it replaces, which is the cleanest available framing of the question this section asks. It comprises over 1,400 real freelance software engineering tasks carrying $1,000,000 in actual payouts, ranging from $50 bug fixes to $32,000 feature implementations, and it scores a model by the dollar value of the work it completes rather than by a percentage. Frontier models at the time of publication earned something over $400,000 of the million available (arXiv).
That result can be read in either direction, which is precisely why it's worth citing: a machine capturing forty percent of the market value of a real freelance software market is a remarkable capability result, and a machine leaving sixty percent of it uncaptured is a measurement of how much of commercially valuable engineering remains outside its reach.
A benchmark of over 1,400 real freelance tasks carrying $1,000,000 in actual payouts, ranging from $50 bug fixes to $32,000 feature implementations, saw frontier models earn somewhat over $400,000 of the total available. (arXiv)
Two conclusions follow, and the second matters more than the first.
The first is terminological: this report will continue to say that code has become cheaper, because measured against the wage bill of writing it by hand it plainly has.
But you should understand the claim as one about a ratio rather than about an absolute approaching zero. You should also understand that the ratio is deteriorating for agentic workloads even as the price per token falls.
The second is that this evidence strengthens rather than weakens the argument of the report. If generation were genuinely free, then generating speculatively and discarding what fails would be a rational strategy and judgment about what to build would matter less.
Because generation carries a real and rising marginal cost, and because the cost per delivered outcome exceeds the cost per attempt by two orders of magnitude, the decision about what to attempt reacquires economic weight.
Cheap-but-not-free generation makes the scarce input (knowing what's worth generating and being able to tell whether the result is correct) more valuable, not less.
A caution belongs here rather than in the appendix. Prices in this market are volatile, competitive, and possibly subsidised by capital seeking market share. Don't read anything in this section as a forecast of what inference will cost in three years.
The direction of the per-token price is downward and has been steeply so. The direction of the per-task cost has been upward, because task ambition has grown faster than unit price has fallen. Which of those dominates in any particular organisation is an empirical question about that organisation's workload. The practical recommendation is simply that engineering leaders should measure their cost per delivered change rather than assuming it's negligible, because at present most can't state it.
5. The Case for AI: What the Positive Evidence Actually Shows
It would be straightforward, and dishonest, to build the argument of this report exclusively from studies showing disappointing aggregate outcomes.
The experimental literature also contains findings of substantial, well-measured, statistically robust gains, and these deserve to be presented at full strength rather than acknowledged in passing. They define the conditions under which AI assistance genuinely does what its advocates claim.
The clearest of these comes from a randomised controlled experiment involving ninety-five professional developers, all of whom were asked to build the same JavaScript HTTP server.
Developers with access to GitHub Copilot completed the task in an average of one hour and eleven minutes, compared with two hours and forty-one minutes for the control group. This is an improvement in task completion speed of 55.8 percent, with a 95 percent confidence interval running from 21 percent to 89 percent. Completion rates also improved, with 78 percent of the treatment group finishing the task against 70 percent of the control group (Microsoft).
In a randomised controlled experiment with 95 professional developers building the same JavaScript HTTP server, GitHub Copilot increased task completion speed by 55.8 percent, reducing average completion time from 2 hours 41 minutes to 1 hour 11 minutes, with completion rates of 78 percent versus 70 percent. (Microsoft)
That is a large effect on a clean experimental design, and it shouldn't be explained away. The characteristics of the task, though, are as informative as the result: the problem was bounded, the specification was clear, the scope was fixed, the codebase was new, there was no existing architecture to respect, no production system to avoid destabilising, no team convention to conform to, no historical decision to understand, and no maintenance horizon over which the choices made would have to be lived with.
These are precisely the conditions under which generation is the dominant cost, and under such conditions the evidence indicates that AI assistance roughly halves the time required.
A second study, examining code quality rather than speed, found improvements as well. Among two hundred and two developers with at least five years of professional experience, those using Copilot were 53.2 percent more likely to pass all ten unit tests associated with the assigned task. Their code showed measured improvements of 3.62 percent in readability, 2.94 percent in reliability, 2.47 percent in maintainability, and 4.16 percent in conciseness, and was approximately 5 percent more likely to receive approval in code review (The GitHub Blog).
The magnitudes on the quality dimensions are modest (low single-digit percentages) but their direction is consistently positive, which contradicts the simplest version of the claim that AI-generated code is uniformly worse.
The unit-test result is considerably larger and suggests that on well-specified problems with clear success criteria, assistance meaningfully improves the probability of arriving at a correct answer.
GitHub has separately reported that Copilot generated approximately 46 percent of the code in files where it was enabled (The GitHub Blog). That figure is a measure of penetration rather than of value, and it sh...
