Generative AI

More Agent Autonomy Requires More Control

More Agent Autonomy Requires More Control

Bring LLM-powered apps from prototype to production

Discover a collaborative platform where teams work side-by-side to deliver LLM apps safely.

Bring LLM-powered apps from prototype to production

Discover a collaborative platform where teams work side-by-side to deliver LLM apps safely.

Summarize with AI

Open this article in your preferred AI assistant.

“Human in the loop” sounds straightforward until you try to decide where the human actually goes.

You can put an approval before every meaningful step. 

  1. Approve the ticket. 

  2. Approve the plan. 

  3. Approve the implementation. 

  4. Approve the pull request.

At that point the agent may execute autonomously, but the work still waits on people all day.

The other extreme is equally unhelpful: give the coding harness access to everything it might need and hope the model knows when to stop.

A Software Factory sits somewhere in between.

During intake, the factory can be told which tickets it is allowed to pick up. Planning can still be collaborative. Only after the plan is settled does the agent move into implementation, followed by automated review and human review before merge where required.

Those stages are useful because they give you several places to constrain the work without somebody supervising every command.


A branch is already a form of control

Some of the useful controls are surprisingly mundane.

We ran into a small version of this ourselves with automated changes through Framer. The instruction was simple: work on a branch, never directly on main. A person can merge and publish later.

Nothing about that rule makes Claude better at editing a page.

It just limits what a mistake can do.


The useful distinction is between giving an agent enough capability to complete the work and giving it authority it doesn’t need. Those aren’t the same thing. A factory might need to read a repository, run tests and call an internal service without being allowed to deploy or change production infrastructure.

Tooling can be scoped too. The factory environment can expose the APIs, MCPs, browser or internal systems approved for that type of work rather than inheriting everything available on somebody’s laptop. The agent doesn’t need to reason about those boundaries as they’re properties of the environment it runs inside.

Moreover, the execution environment can enforce another layer of control. A run can be isolated in a sandbox, given only the secrets it needs and restricted by network connectivity policies that determine which external or internal systems it can reach.

It’s also possible for the approval boundary to be narrower than the whole run. An individual MCP tool can be marked as requiring approval: With the agent configured to respect tool-level approvals, that action can pause for a decision without putting every tool call behind the same gate.


Intake gives you another control point.

Maybe the factory scans Jira but only picks up issues assigned to it. Maybe a particular label means the ticket can continue into implementation. A badly specified ticket stays where it is rather than leaving the coding agent to decide whether the requirements are good enough. 

None of these controls depends on the model deciding to behave.

Not every boundary needs a human either. Runtime constraints can put explicit limits on the number of reasoning iterations and the model-execution time available to a run: those limits sit in the execution configuration rather than relying on the model to decide when to stop.

Budgets put an economic boundary around the same autonomy. A factory might be capable of continuing to reason or call tools but doesn’t necessarily mean it should be allowed to consume resources indefinitely. Setting a budget gives management a hard limit on what the factory can spend before the run has to stop or come back for attention.


There are good reasons to make those boundaries explicit.

Stack Overflow’s 2025 Developer Survey showed that 87% of respondents said they were concerned about AI-agent accuracy and 81% about security and privacy.

The awkward part comes after the code works

Implementation finishes and the tests pass.

Another model can still review the change before anybody looks at it. If that review fails, the work goes back to implementation. If it passes, the pull request moves on.

That sounds slower than simply letting a capable coding agent finish the job. Early on, it probably is.

But the friction tells you something.

It shows where the factory keeps getting things wrong, which checks rarely catch anything, and where a human is approving essentially the same safe change over and over again.

Then some of the gates can go.

If humans are approving around 80% of the pull requests from one factory, you probably still want them looking. If that gets close to 99%, there is a reasonable conversation about whether every PR still needs the same checkpoint.

We wouldn't turn 99% into a rule. A database migration and a cosmetic fix can have the same acceptance rate and still deserve very different treatment.


The cost of being wrong matters too

A pass rate only tells you how often the workflow gets through successfully.

What it won’t tell is what the unsuccessful case looks like.

A cosmetic change that slips through with the wrong spacing is annoying, though it can be reverted. 

A migration that corrupts customer data is a very different failure even if both workflows have succeeded 99 times out of 100.


That changes where the controls belong.

For low-risk, reversible work, a team may be comfortable letting the factory run further before anybody intervenes. The same team might keep an approval around a production change, a destructive tool call or anything that crosses a boundary that is difficult to undo.

This is also why autonomy shouldn’t be reduced to a single percentage on a dashboard. Two factories can have identical acceptance rates and still deserve completely different operating rules. 

One autonomy level probably won’t survive contact with the backlog

A factory handling cosmetic bugs could eventually go almost straight from intake to pull request.

Another factory working on a core service may still stop after planning even if both use Claude Code underneath. 


That unevenness is the point.

Autonomy is better treated as a spectrum that moves over time, with different kinds of work moving at different speeds. Getting part of the engineering workload to a useful level of autonomy already has value.

You don't need to wait for the entire SDLC to run without people. You just need to move the boundary where the evidence and the cost of being wrong justify it.

If you’re working out how much autonomy to give coding agents without giving up control, see how Orq.ai helps engineering teams build those boundaries into a Software Factory.



Sohrab Hosseini image

Sohrab Hosseini

Co-founder (Orq.ai)

About

Co-founder of Orq.ai. Previously led and grew SaaS companies as COO/CTO and worked as a McKinsey associate.

Create an account and start building today.

Create an account and start building today.

Create an account and start building today.

Create an account and start building today.