“Add account export” sounds like a normal engineering ticket, and for an engineer who already knows the system that’s probably enough. They know there’s an export service, that internal audit records were deliberately excluded from customer exports and that billing history comes from another system.
They might also remember why an earlier version of the feature was rejected, even if there’s not much left in the code to tell you.
An agent can find the export service, follow the calls and see where billing lives, then build something that looks completely reasonable from the repository. But if the ticket never says that audit records were intentionally left out, or that an earlier approach was already ruled out, there may be nothing in the repository that tells it otherwise.
What is spec-driven development?
Spec-driven development is basically about deciding what shouldn’t be left for implementation to invent.
GitHub’s Spec Kit splits the work into Specify → Plan → Tasks → Implement → Converge. Technology choices come later, after the specification has set out what the work is supposed to achieve. The agent gets the outcome and the important constraints first, then works out how to build it once it has seen the repository.
In a Software Factory, that specification becomes the contract the rest of the run works from. Planning, implementation, evaluation and approval can change how the work gets done without silently changing what the task was approved to achieve.
A file-by-file plan written that early can be wrong as soon as the agent opens the repository.
The agent might find an existing helper that removes half the planned changes, or follow the calls and discover that the behaviour actually belongs somewhere else. There’s no reason to make the agent stick to the original plan just because somebody guessed at the implementation earlier.
You do want to settle the things that would change what the task means. If billing history is supposed to be part of an export, say so. If a public contract can’t change, say that too. The agent can move code around, reuse something you didn’t know existed or take a different route through the repository without also getting to decide those things for itself.
Why normal tickets break down with coding agents
Meta ran into a version of this problem in one of its data pipelines. The system spans four repositories, three languages and more than 4,100 files, but agents were still missing conventions engineers knew and the code didn’t make obvious.
They ended up documenting more than 50 of those patterns and generating 59 compact context files. In preliminary testing, agents using them needed roughly 40% fewer tool calls and tokens.
“Use the existing export service” could be ordinary implementation advice or it could be there because the service already applies an authorization or privacy check. If it was only an implementation suggestion, finding a better route is fine. If the service is carrying a privacy or authorization check, ignoring it is a different problem.
How detailed should a spec be?
We don’t think a spec should turn into an implementation plan written before the agent has even opened the repository. A small internal refactor with reliable tests often needs little more than the intended behaviour, a few constraints and a way to tell whether it’s safe to merge.
Writing two pages of requirements for it can leave somebody reviewing details the agent is about to discover for itself anyway.
File names and test locations often look obvious until somebody actually follows the code.
A one-line configuration change can alter customer-facing behaviour. A large internal refactor can move hundreds of lines without changing a product decision at all.
Shopify’s Helix project is moving its mobile app from React Native to native Swift and Kotlin, but the migration still moves through smaller checkpoints, quality gates and engineer review rather than trying to describe the entire job upfront.
What belongs in the task, and what doesn’t?
If sensitive work already has an organization-level policy, copying the whole thing into every spec gives you another version to keep current. For a task involving customer records, the spec can carry the parts that matter to this run, such as a privacy review before merge and a fixed budget.
Our view is that the task should carry what changes for this run, not become another copy of the policy system around it.
A ticket can look like ordinary application work until the implementation reaches something nobody had to decide when the task was written. The service needs another environment variable, or a setting that looked local turns out to change the deployment as well.
None of this had to be decided when the ticket was written, but it does now.
UnderSpecBench tested DevOps tasks where the intended safe action stayed the same while details about the target, intent and blast radius changed. Across five agent-model configurations, 55.8% to 67.8% of acted runs crossed at least one action boundary.
Those numbers come from DevOps tasks, not a general benchmark of coding-agent failures. In those tasks, missing information wasn’t always enough to make the agent stop.
You can write “don’t exceed this boundary” into a spec, but that sentence won’t stop anything by itself. If crossing it should halt the run, send the work back or trigger approval, the workflow has to do that.
What happens when the spec changes?
The approved spec calls for a new preference store.
Halfway through the work, the agent finds a shared service that already handles the same data and exposes most of what the new store would need. Reusing it would save another migration and another storage path, so the original plan suddenly looks a bit wasteful.
The shared service handles the data in a way the original task didn’t account for. Now the agent isn’t just choosing a cleaner implementation. It found something that affects what was approved in the first place.
A stale path or missing test reference can be corrected as the work moves, and a privacy check that turns up an unexpected field is different. It can change what has to happen before the feature ships.
“Living specification” is a useful phrase until very different kinds of changes start getting grouped together. Discovering that the preferred implementation changes how customer data is handled can pull another team back into the task.
We think that distinction is more useful than treating every edit as a “living spec” update. An agent can correct what it learned about the implementation without automatically getting authority to change what the team approved.
How do you verify a spec-driven task?
Passing the application tests only tells you that the implementation behaves the way the tests expect. An export can still include a field that was meant to stay internal, or a refactor can leave every test green while changing a public contract the task said not to touch.
The finished work needs to be checked against the approved task as well, including any changes made while the run was in progress.
A failed acceptance test can send the work back for another pass. An unresolved data question needs a different response.
You need enough of the run history left behind to see why the work stopped and what still needs an answer. For us, that’s part of what makes the specification useful after execution starts.
It gives evaluations, approvals, and reviewers something stable to compare the run against rather than treating the final diff as the whole story.
If you want to see how Orq.ai handles that workflow in practice, book a demo.
