From Backlog to Production

A Software Factory doesn’t start when an agent begins writing code.

In the workflow we’ve been thinking about, work can be picked up from Linear, Jira or GitHub Issues, triaged, planned, implemented, reviewed, released and then fed back into the system.

The coding agent is only doing one part of that.

• A ticket can be assigned to a factory and still be unusable. 

• A good implementation can still fail QA. 

• A run can technically succeed while exposing something wrong with the context, tests or tooling around it.

That distinction is showing up beyond individual workflows. 

DORA’s 2025 research found that greater AI adoption was associated with higher software-delivery throughput but still with lower delivery stability. Faster implementation can expose weaknesses further down the path rather than remove them.

The workflow has to deal with all of those cases without turning every one into another prompt for an engineer.

Intake decides what enters the factory

A factory can watch the systems teams already use and pick up work assigned to it. The first version of the rules might be nothing more exotic than the right label, enough information in the ticket and a few requirements that have to be present before work starts.

We’d expect teams to get some of those rules wrong.

bug is a good example. It sounds like a useful category until you put a few tickets next to each other. Fixing the wrong validation message and investigating intermittent writes against a shared service can both arrive as bugs. 

One might be routine. The other may not even have a settled diagnosis yet.

After a while, the team might stop routing the entire queue through one factory. Perhaps it takes bugs from two services with good tests and well-understood failure modes, while everything else stays outside.

That’s enough. Intake doesn’t need to produce a grand taxonomy of engineering work.

Planning is where the ticket usually gets less tidy

Take a small API request:

That sounds reasonably executable until the repository is opened.

The response comes from a shared schema. One consumer still assumes the old shape. Another transforms it before persistence. The test most likely to catch a regression isn’t even in the package being changed.

An engineer who works on the service may know half of that already. 

The ticket writer probably didn’t think it was worth spelling out. The factory has to get there another way: repository inspection, tests, documentation, Skills and whatever other engineering context has been made available to the run.

This is also why planning can be collaborative rather than a prompt-expansion step. The flow allows a person to look at the proposed plan before a longer implementation run begins.

Sometimes there’s nothing interesting to decide. The plan is obvious enough and the run can start. Other times the codebase exposes a choice that the ticket never resolved. 

There may be two technically valid implementations and nothing in the task that says which one the team actually wants. Someone answers that question and the run continues.

A different ticket might stop here completely. We’ve learned something useful even if no code is written: this piece of work wasn’t ready after all.

Implementation happens in an environment built for the work

The factory needs somewhere to check out the repository, edit code and run tests. Depending on the task, it may also need an approved API, a headless browser or another internal system exposed through an MCP. The harness can run in its own environment with those tools available to it.

A test fails in another package. The agent traces it. A dependent change is needed. It makes that change and runs the checks again.

Some runs will last a few minutes. Others won’t. Longer runs are where a real execution environment starts to matter. An engineer shouldn’t have to keep reopening the task just to translate the latest error into another instruction.

What we care about more than the agent’s summary at the end is what the run left behind. 

Review starts before a human opens the diff

The factory flow can put automated AI review between implementation and human review.

That review can be part of the agent configuration rather than something a developer has to start manually. For example, an evaluator can run against the agent’s output:

Illustrative evaluator ID; the configuration follows Orq.ai’s documented evaluator settings.

The important part here is where the check sits. It can produce a signal before a human opens the diff, and that result can determine whether the work continues or goes back around.

A lot of what gets caught here is fairly boring. The implementation skipped something in the plan. A known rule wasn’t followed. A test that should’ve changed didn’t.

Good. 

Fix it before somebody spends time reviewing the same mistake.

We wouldn’t turn every finding into a new instruction for future runs, though. One strange implementation choice may just be one strange implementation choice. Reviewers disagree too.

Repeated findings are different. If the same comment keeps coming back, the review result should feed into whatever produced it: 

• Planning

• Guidance

• Context

• Tests

• The implementation setup itself

Human review still sits in the path where teams require it. That path doesn’t have to be identical for every change.

QA and release have their own problems

The PR can look fine then an integration test uses an older payload and fails. 

There’s no contradiction there. The implementation tests proved what they were designed to prove.  They just didn’t cover that case.

One kind of maintenance work may only need a larger automated suite before release. Another workflow might need a representative environment, another service to be ready, or an explicit approval before deployment.

We don’t think there’s much value in forcing all of that into one universal release template. The conditions should follow the work.

And sometimes the workflow moves backwards. QA exposes a missed dependency, so implementation resumes. A release check shows that the plan assumed another service was already migrated, so somebody has to revisit the assumption.

After release, the useful signal is usually recurrence

Most things that go wrong shouldn’t result in another factory rule.

A production issue appears. The team fixes its and perhaps the failure was unusual enough that nothing else needs to change. The situation looks different after the third similar failure.

Several changes break against the same older customer configuration. At that point we’d rather add the case to QA than keep fixing it after release.

Other patterns point somewhere else. 

• Review keeps correcting the same implementation choice: inspect the guidance available to the run. 

• An MCP fails across unrelated tickets: fix the integration. 

• A class of work keeps reaching planning only to be rejected: intake is probably too broad.

The useful part of the feedback loop is knowing where the correction belongs.

These changes won’t arrive in a nice order. A team may tighten intake and later discover the bigger problem was stale architecture guidance. Another workflow may spend months exposing weaknesses in tests or tooling that nobody noticed when humans were compensating for them manually.

Some parts of the factory will eventually become boring and stay that way. 

Others will keep getting pulled apart and rebuilt.

The point of running the whole path from backlog to release is that you can finally see which part of the system needs to change.

If you’re building the workflow around the coding agent and not just the coding agent itself, book a demo to see how Orq.ai helps engineering teams turn the path from backlog to production into a governed Software Factory.