AI writes the code. Experience writes the ticket.

We’ve been running AI coding agents inside our production engineering work for a while now, and the most useful lesson so far is an unglamorous one: what matters is the ticket.

Give an agent a vague instruction, and you get plausible code that closes the ticket. Give it a specification that names the acceptance criteria, the architectural decisions to respect and the behaviour that must not change, and the same model produces code that ships. The difference between the two runs is twenty years of knowing what to put in the ticket.

The generation step is fast and getting faster. What’s happened is that the expertise has moved. It used to live in the keystroke, and the senior engineer was the person who could write the tricky bit. Now it lives upstream and downstream of the keystroke: in specification, which is knowing what the system has to do and what “done” means, and in verification, which is knowing whether what came back is true. Neither of those improves when the model does.

Here’s a real one: the opening ticket from a live engagement with a creative technology partner, lightly anonymised.

Read it again as an artefact rather than a task. Almost every line has a specific job.

Every acceptance criterion can fail. There isn’t a single quality adjective in the list, because “robust”, “performant” and “well-tested” can’t fail a review, since nobody can say what failing would look like. “Creates a `render_jobs` row at status `submitted`” either happened or it didn’t. A checklist of falsifiable statements is the only kind an agent can be measured against, and the only kind a human reviewer can be held to either.

The ticket names the decisions it must respect. ADR 027 established preview renders as a ticket-scoped concept, and ADR 016 defines the production render model. Naming both in the first paragraph is what stops the implementation from “helpfully” merging them into one elegant abstraction. Why not let the model unify them? Because from inside a single ticket, unifying two similar-looking things is what good code looks like. Here it’s the wrong call, and the only way to know that is to have been in the room when 016 and 027 were written.

It names what must not change. “Existing queue behaviour is unchanged.” “Per-creative RBAC is unchanged.” “ADR 016 is untouched.” This is negative space, the hardest thing in software to specify, and a model will never infer it, because the information isn’t in the repository. The reason the queue behaves as it does may be an incident from eighteen months ago; the reason RBAC is scoped per creative may be a contractual obligation to a client who appears nowhere in the code. It’s a bit like asking a builder to fit a new kitchen without mentioning which wall is load-bearing – they’ll find out eventually, but you’d rather they found out from you. If something must stay as it is, the ticket has to say so.

UI, endpoint, job dispatch, database, external service client, object storage, and back to the UI, with nothing mocked – that’s the walking skeleton, thin but complete through every layer. Building this shape first means the integration risk (the render service’s actual REST contract, the storage key namespacing, the status mapping, the RBAC boundary) gets discovered on day one, when it’s cheap to be wrong, rather than in week six, when four other tickets have been built on an assumption that turns out to be false.

And the small clauses carry real weight. “Named columns rather than positional access” is a decision about what happens when someone adds a column in eight months. “Configuration read lazily from the environment” is a decision about not compiling in a fallback and then discovering in production that the fallback is what has been running all along. And a pure mapper with no I/O means the layer-name join can be tested without a network. One line each, and each one is a year-three cost avoided in the first week.

The vague version of this ticket exists, and everyone has seen it:

An agent given that will produce something, and it’ll probably look good. It will also, very likely, reuse the production render path, because that’s the render path that exists. It’ll write the preview PNG wherever the nearest example wrote its files, inheriting or bypassing whatever permissions that location carries. It’ll hold configuration in a module-level constant with a sensible default so nothing crashes on import, access rows positionally because that’s what the surrounding code does, and produce a test asserting the endpoint returns 200.

None of that is a model failure. Each choice is a defensible reading of the instruction it was given. The specification didn’t distinguish preview renders from production renders, so the implementation didn’t either.

There’s a decent measurement of how common the vague version is, and it comes from an unexpected direction. When OpenAI built SWE-bench Verified, they had 93 professional developers review the 1,699 real GitHub issues the benchmark was drawing on. 38.3% were flagged as underspecified, meaning the problem statement didn’t say enough for a reviewer to tell whether a given fix was correct. Those tasks were removed, because a benchmark full of them measures the ticket rather than the model.

These are issues from well-run open-source projects, written by people who care about the repository, and nearly four in ten of them can’t be graded. Nobody wrote them badly on purpose. They were written for a reader who already knows the system, which is fine for a human colleague and useless for anything else. Your backlog is probably in much the same state, and ours was.

Specification covers the front end of the problem. The back end is knowing whether what came back is true, and this is where most teams are quietly exposed. What we watch for is the vacuous test, one that passes regardless of whether the code is correct.

Here’s a concrete one, from a real suite. A test asserted that a generated storage path matched an expected storage path. Both were built from an entity ID. The ID passed in didn’t resolve, so both sides of the assertion evaluated to an empty string. The assertion was true, and it stayed true for any implementation, including one that did nothing at all. It had been green for months, counted in a coverage figure people were reading as a measure of safety. The question nobody asked when it was written is the one we now ask of everything: if the implementation were wrong, would this test say so? Here it couldn’t, because the assertion compared a value with itself.

No suite is at least honest about what you don’t know, while a green suite that asserts nothing sells you confidence you haven’t earned, rather like a smoke detector with the battery taken out. This class of test is also the easiest thing in software to generate at volume, because it satisfies every visible signal: syntactically valid, names the function under test, runs fast, green.

So we hunt them. The question we ask of any test, human-written or generated, is what change to the implementation would make it fail. If there’s no answer, the test is decoration. Assert on real values rather than truthiness, and on state after the call rather than the return code. Make the test fail once, deliberately, before you trust it.

This matters more than it did three years ago, because the feeling of correctness has detached from correctness. METR’s randomised trial with sixteen experienced open-source developers across 246 real issues is the sharpest illustration around: developers expected AI to make them 24% faster, were in fact 19% slower, and afterwards still believed they had been 20% faster. METR are careful to say this is a snapshot of early-2025 tooling in one setting, and that’s how it should be cited. What survives the caveat has little to do with speed. The practitioners’ perception of their own output was wrong, in a measurable direction, and it didn’t correct itself on contact with reality. Which is why verification has to be something written down that a reviewer can check.

The teams getting compounding returns from AI have one thing in common, and it has little to do with which model they can access. A unit of work arrives at the generation step already carrying its acceptance criteria, its architectural constraints, its explicit non-goals, and a test harness that would fail if the code were wrong.

Long story short, the bottleneck we keep running into is the specification, and the specification gets written by whoever has seen this system, or one close enough to it, break before.

If you want better output from the tools, I’d start with your tickets rather than your subscription tier. It’s the cheaper fix, and it keeps paying off on the days no agent is involved. If you’d rather have someone go through the backlog with you, that’s part of what our Production Readiness Review looks at – ping us.