A Software Factory gets busier long before software delivery gets better.
Agent runs and pull requests are easy to count. So are tokens, tool calls and generated code. Once agents take on more implementation, most of those numbers should rise anyway. What’s more important is what changed outside the agent.
DORA’s delivery metrics give us part of the answer since they sit further downstream. They weren’t designed around coding agents, which is useful here but they also leave some obvious gaps.
DORA won’t tell us what autonomous execution cost or how much work got through without unexpected human intervention.
DORA catches what agent activity misses
Anthropic has already run into an extreme version of this. In May 2026 it averaged around 2,500 merged code changes a day. By September that had risen above 30,000.
Coding itself stopped being the bottleneck as source control and CI became the problem instead, to the point that Anthropic built its own source-control system around the new workload. Implementation throughput went up enough that infrastructure designed around slower human production became the constraint.
DORA’s recent AI research has been getting at the same issue from a broader angle. Its 2025 State of AI-assisted Software Development research described AI as an amplifier of the engineering system around it.
Its 2026 analysis found some of the time saved during generation shifting into auditing and verification, while higher AI adoption was associated with increases in both throughput and instability.
The current DORA framework has five delivery metrics:
Dimension | Metric | What it tells you |
Throughput | Change lead time | How long a committed change takes to reach production |
Throughput | Deployment frequency | How often changes are deployed to production or released to users |
Throughput | Failed deployment recovery time | How quickly service is restored after a failed deployment |
Instability | Change fail rate | How often production changes require immediate remediation |
Instability | Deployment rework rate | How often deployments are unplanned work needed to fix production problems |
Faster coding can expose a slower system
Change lead time is probably where the shift becomes visible first.
If implementation drops from hours to minutes and lead time hardly moves, generation speed isn’t the most useful thing you should try to optimize anymore. Review might account for most of the elapsed time. An approval queue that was tolerable when implementation took a day can look very different when the change itself took ten minutes.
Deployment frequency gives another view since a factory can produce substantially more pull requests without substantially more software reaching users.
There’s evidence this isn’t limited to unusually agent-heavy AI companies, either.
Harvard researchers studying 300 million engineering work events across 718 firms found coding assistants and agents increased coding productivity yet those gains didn’t fully pass through into software output. For AI agents, review took longer, more updates needed revision, and reviewers left more comments.
Exactly why that happens is harder to separate cleanly and there isn’t a clear answer we can give. More generated code reaching review can itself create more review work.
Either way, implementation productivity and delivery productivity aren’t interchangeable.
Failed deployment recovery time sits somewhere else again, as agents can inspect logs, search repositories, prepare patches, and run tests during an incident.
A factory could become valuable there without changing the normal feature-development numbers much at all.
Then there’s what happens after production
First-pass implementation speed tells us even less once the change ships.
Change fail rate looks at production changes that need immediate remediation. Deployment rework rate (which DORA added in 2024) captures deployments happening because something already in production needs fixing.
An agent can write a change quickly, get it through review and have it deployed that afternoon. If an emergency patch follows the next morning, the original implementation time is still accurate.
It’s just not much of an account of the total work.
For a factory, that downstream rework has to remain visible even when everything upstream looks more productive.
The numbers change when the work changes
There’s a problem with comparing any of these metrics over time: a factory rarely keeps handling exactly the same work.
A dependency update and a cross-service migration can each end as one merged pull request. The infrastructure may be the same but the engineering work clearly isn’t.
DORA recommends applying its metrics in the context of an application or service. Something similar matters when measuring a factory.
Look at the engineering workload first
It helps to know where engineering time went before automation started moving it around.
There might be:
Vulnerability remediation
Dependency upgrades
Routine bugs
Difficult production issues
Frontend work
Migrations
Larger product changes
What matters is being able to tell whether a factory has taken over a meaningful slice of work or simply produced a high autonomy number on a small collection of convenient tasks.
If repetitive maintenance used to consume a noticeable share of engineering capacity and now mostly runs through a factory, there’s a baseline to compare against.
“60% autonomous” across an undefined set of engineering tasks gives you much less.
There’s also an engineering decision hiding before the measurement starts, because somebody has to decide which work is worth turning into a factory workflow in the first place.
Frequency matters and so does whether the input can be recognised and the outcome checked. Some work will remain too ambiguous, too risky or too rare to bother with yet.
A factory can expand in two directions
One workflow can become more autonomous, or more kinds of work can become suitable for factories.
Those aren’t quite the same measure of progress.
An enterprise could keep human approval around every production merge while gradually moving dependency remediation, test repair and a narrow class of bugs into factory workflows.
Its autonomy percentage may not look spectacular.
A meaningful amount of routine engineering work may still have disappeared from manual execution, which makes coverage interesting even if it never becomes a neat KPI.
Work backwards when a number moves
If lead time rises, then review latency and CI duration become worth looking at. If change fail rate changes, eval results or reviewer corrections might explain where things started going wrong.
A spend increase could come from retries, tool failures or a different model mix.
If the same class of run keeps drifting in the same place, perhaps the factory needs different instructions or the relevant architecture context isn’t available.
What did the change cost?
Let’s say attributable model and runtime spend comes to $8,000 for a month and those runs produce 200 merged changes.
That’s $40 per merged change.
AI spend per merged change = attributable model and runtime spend ÷ agent-attributed merged changes
Uber is already measuring its Software Factory in similar units. For managed agents, it tracks outcome-denominated costs including cost per merged PR, cost per review, cost per alert and cost per cleanup.
By August 2026, more than 70% of Uber’s pull requests were attributed to local or cloud agents, with more than 30,000 agent-skill executions a day.
Its model evaluations follow the same logic. For code review, Uber compares cost per review with precision, recall and F1 against known bugs instead of assuming the model with the cheapest tokens produces the cheapest useful result.
Our $40 figure has an obvious weakness, though.
A dependency bump adds one to the denominator, and so can a multi-service change that needs much more verification.
That doesn’t make the metric useless. Comparing the same workload with itself is much more convincing.
If dependency upgrades move from $18 to $11 per merged change, it could come from:
Fewer retries
Better routing
Better context
A different model
Trying to make that $11 directly comparable with a migration is harder to defend.
There’s probably a point where normalizing all of this creates a more sophisticated metric without giving us a better one.
Verification cost is harder to place
We wouldn’t put human labour into the headline number at first. Review effort clearly matters.
It may matter more, particularly as implementation gets cheaper.
If an engineer spends 25 minutes reviewing an agent-produced change, some of that cost may be attributable to the agent and some belongs to the task regardless of who wrote it. A human-written implementation would probably have needed review too.
Fold all of that into cost per merged change and the number starts depending on salary assumptions and questionable attribution. Leave it out completely and a factory can look cheaper while engineering time moves into verification.
We’d start by keeping execution cost and verification effort visible separately. Whether they eventually belong in one number probably depends on what starts happening to the review workload.
Autonomy depends on what the factory was expected to do
Stripe’s Minions make the autonomy problem pretty obvious.
By February 2026, more than 1,300 Stripe pull requests a week were being merged with all of the code produced by Minions and no human-written code in the change, but humans still reviewed the pull requests.
Is that autonomous?
At the implementation level - clearly.
End to end - no.
More importantly for measurement, the human review was expected. Nobody had to step in because the agent lost its way. That’s different from an engineer rewriting the plan halfway through a run or taking over the implementation.
Planned approval isn’t the same as intervention
Autonomous completion rate = percentage of eligible Software Factory tasks completed from assignment to accepted result without unplanned human intervention in execution
Planned gates shouldn’t count against the agent for reaching exactly the boundary the workflow gave it.
Unplanned intervention is something else:
The plan needs rewriting
Somebody manually corrects the implementation
The run goes off course and has to be rescued.
There’ll be less obvious cases like somebody changing two lines during review or the agent asking for clarification.
Different teams may classify some of those differently which is probably unavoidable.
The definition at least needs to remain stable enough that improving the metric doesn’t become an exercise in moving the line.
“Eligible” does most of the work
A routine dependency upgrade might be suitable for substantial autonomy while an authentication migration isn’t.
That makes comparisons like “Team A is 65% autonomous and Team B is 40%” difficult to interpret. The percentage only tells us something once we know which work each team considered eligible in the first place.
The workload map starts mattering again here.
Can low-risk cosmetic bugs run from assignment to accepted change without somebody stepping in? What about routine dependency remediation?
A production issue affecting authentication is a very different workload.
One factory reporting 40% autonomous completion across difficult production work may be further along than one reporting 65% across simple maintenance tasks.
An 80% target is easy to game
We wouldn’t set one.
Give the factory easier tasks, shrink what counts as eligible, and lower the acceptance bar. The autonomy rate improves without the factory necessarily getting better.
A factory doing 50% of genuinely useful production work autonomously may be doing considerably more than one reporting 90% across work that was easy to delegate anyway.
Human gates can disappear at different times
The more useful signal is what people are still changing.
If reviewers regularly reject or substantially modify a particular class of changes, that checkpoint’s doing useful work. Later on, it probably stops catching much. Plan approval might disappear for that workload while PR review stays.
There’s no reason another value stream has to progress in the same order.
The engineer’s job changes before the metrics do
Once more execution moves into factories, an engineer can become more useful while personally producing less visible output.
They might identify which workload is worth automating next or notice that a class of runs keeps failing for the same underlying reason and change the context or workflow behind it.
Their own pull-request count can fall while more engineering work gets through the system, which is why these are factory metrics in the first place.
A personal token count won’t explain much about it either.
Keep in mind that measurement is only useful if it changes how the factory works. The next step is connecting delivery outcomes back to agents, evaluations, controls, and workflows producing them.
If you’re working out how to measure and scale agentic software delivery, book a demo to see how Orq.ai fits into that system.
