Verification Is the Bottleneck Once Agents Write the Code

In a video where Kun Chen builds an app through an orchestrator agent, the agent reports that both pull requests are merged and the app passes validation in the simulator. His reply is "I need some evidence." One question later the agent admits the app is not connected to the backend at all. It is running on mocks. The merge was real and the build was green, and neither fact said anything about whether the thing worked.

In a recent pass over my research library, 98 videos from 46 creators, fourteen sources made some version of the same claim without citing each other: writing code got cheap, and confirming it is correct did not. John Kim puts the arithmetic plainly. One person can now run several agents, so reading all of your own output is no longer practical.

Something has to replace reading every line. Across the notes I see three answers, and they are worth more than the slogan.

Gates that do not need the model's cooperation

Caleb Curry's version is the cleanest. An instructions file is interpreted by the agent, so it is a place for instruction and a poor place for enforcement. He says his biggest change was noticing that rules written in markdown got inconsistent compliance. His example is tenant isolation: the instruction was to respect tenant scoping, and the enforced version is a database adapter that attaches the tenant to every query, so forgetting is not possible.

The cheapest gate he describes is a ratchet on lint violations, run from a hook when the agent finishes responding.

baseline = violations recorded at the last passing run
if current > baseline: fail
if current < baseline: baseline = current

Old violations are grandfathered. New ones fail. That makes it adoptable in a project that already has a messy history.

Robert Martin, interviewed by Matt Pocock, reaches the same place from a different start. He wrote a long document on how code should look and found the model treated it as guidelines. His explanation is that long instructions get pushed into the middle of the context and lose weight, though he describes his own research on that as thin. His response was to cut the prompt down and run deterministic tools afterward.

Two honest costs. Martin says gates trade speed for quality: by his figures, a task one agent finishes in five minutes takes about an hour through his full pipeline. And Curry's talk shows no code or repository, with setup deferred to a paid course. His claim that the whole stack makes high quality very likely is asserted, not shown.

I have one gate of this kind that I trust. My yt-research skill writes notes whose claims carry timestamp anchors, and it refuses to finish until those anchors verify. It does not ask the agent to cite carefully. It fails. By Curry's test, though, a fair number of my other standing rules still live as prose in an instructions file, which makes them suggestions.

Review depth scaled to blast radius

John Kim's procedure treats a codebase as a tree. Trunk code is entry points and shared infrastructure, where a defect reaches everything. Leaf code sits at the edges. His rule is that review depth follows the blast radius of the change: he skims leaf code for anything obviously wrong, and an ungated change to shared state gets real reading time.

The part that makes this work happens before any code is written. He decides at planning time which pieces touch nothing existing and which are integration points, then puts the integration points behind a feature gate. Gated code counts as leaf code. The question that licenses skimming is whether the change can be rolled back.

Poteto, another of Pocock's guests, says something compatible about irreversible decisions. Where work is programmatically verifiable, a one-way door becomes a two-way door. Where it is not, he does not claim to have an answer.

The limits are worth stating. Kim's examples come from a small demo app, not production code, and he concedes real feature gating needs infrastructure. Most of my own repositories have neither a feature gate nor a canary, so I do not yet know what trunk-and-leaf degrades into there.

A reviewer that did not write the code

Kim is explicit that the reviewing agent must be a fresh sub-agent without the authoring context. An agent that carries the conversation that produced the code will anchor on it and, in his word, "cheat". Curry builds the same separation for a different reason: rules that cannot be scripted, like spotting n-plus-one queries, go to a verifier subagent with its own context, and pulling everything scriptable out is what keeps that verifier focused.

The weakness is that the fresh-reviewer claim arrives without a demonstration. Kim offers no case of a clean reviewer catching something the author missed. He also says AI already reviews better than most humans, and separately that he reads more code than he used to, and does not square the two. The mechanism is plausible, and it argues against the easy move of asking the same session to check its own work. It has not been shown here.

What nobody measured

Almost all of this is practitioner judgment. No source in the set measures what the verification apparatus costs against the time it saves. Martin says there must be a point where the gates slow agents below human speed, and that he has not found it.

I still treat the premise as a working one, because the sources arrive at it by different routes and Kun Chen's is a failure caught in the act, not an argument. The test I apply to a workflow now is where enforcement lives. A rule in a prompt will decay. A rule in a script, a hook or a separate reviewer has a chance.

Key Takeaways

A green build and a merged PR are reports. Ask for behavior you can observe, and name it before the work starts.
If a rule matters, move it out of the instructions file and into something that fails.
Spend review time by blast radius, and use gating and rollback to shrink the radius.
Review with a context that did not write the code. Plausible, widely repeated, not yet demonstrated in these sources.
Nobody has measured the cost of all this checking against what it saves.

Sources

Next step for me is small: pick the rules in my instructions file that I actually care about and turn each one into a script that fails.