Checking the Evidence That the Harness Matters More Than the Model

"The harness matters more than the model" has become one of those lines people say as if it were settled. I am inclined to agree with it. My main agent harness is Claude Code, and I maintain a library of skills around it.

Being inclined to agree is a reason to check. So I went back through the sources in my notes and sorted each claim by whether it was shown or only asserted.

One limit applies to all of it. My notes were built from automatic captions. The charts in the conference session are known to me only through what the speakers said about them, and I reproduced nothing.

What the claim compares

Google Cloud's short explainer is the clearest definition I have. The harness is the software around the model, and it has four design areas: context and memory, tools and execution flow, runtime security and privacy, and evaluation. The video makes no performance claims at all. It is a vocabulary, and a useful one, because it shows how much sits on the harness side. The slogan sets one variable against nearly everything else in the system.

The bare version of the slogan shows up in an IBM explainer from August. The presenter says benchmark gaps between top labs have narrowed, so recent gains mostly come from better harnesses. No evaluation is cited. He also grants that models improved with every release, so the claim is about a share of the gain, and nobody measures the share.

The strongest evidence is one benchmark

The best-supported source is a Y Combinator paper club session titled with the slogan itself. The host opens with an 18 percent gap between two harnesses. He names no source and no task, which makes the headline number the weakest one in the hour.

The real evidence is ARC-AGI. As the session reports it, the best verified score on the private holdout was about 30 percent, and Seth Karten's harness reached 95.5 percent with what my notes record as the same model. That is the right shape of evidence: one model, two harnesses, a benchmark built to resist memorization.

Karten also volunteered things most presenters would not. His first run scored 99.9 percent, and he found in the logs that it was cheating, so he rebuilt the sandbox. He ran the comparison under Claude Code himself, got poor results, and reported the published numbers instead. On other benchmarks he claims only parity or slight improvement. On a week-long research task he says no benefit could be attributed to the harness over the model.

It is still weaker than it sounds. It is one benchmark. The run used a system prompt taken from a community leaderboard, so part of the gain is someone else's prompt work. And every number was presented by the person who built the system, at an event hosted by an investor in some of the projects.

Why the head-to-head cannot carry it

The evidence most people will actually see is a different kind: a creator gives two coding harnesses the same prompt and declares a winner. Nate Herk did this with Codex and Claude Code, asking each to build a form-product clone, and reported one as 11 times faster and 6.6 times cheaper.

By his own disclosures, which are to his credit, the comparison cannot support that.

Each tool ran once. He compares it to pulling a slot machine lever.
The model stacks differed. One side ran a single model throughout. The other ran three, and the orchestrator was not the model he had started the session with.
The cost figures are API-billing estimates, not money spent, and two different numbers appear for the same run, $832 and about $447.
One of the two tools under test served as the judge, working from his summaries.

The second point is the one that matters for this claim. If the harness and the model both change between runs, the experiment cannot say which one did the work. A head-to-head between products compares bundles. He also says, more than once, that the outcome depended heavily on how well the prompt fit each tool.

What survives is an observation. Under the same instruction, one run used 35 sub-agents and about 2,800 tool calls, and the other used 126 sub-agents and about 32,500. Differences that large are not noise. They show that these systems behave very differently. They do not show which layer is responsible.

Claim	Who	Status
18 percent gap between two harnesses	YC session host	Asserted, no source or task
About 30 percent versus 95.5 percent, same model, ARC-AGI	Host and Seth Karten	Reported with caveats, one benchmark, presenter's own system
Most recent gains come from harnesses	IBM presenter	Asserted, no evaluation cited
11 times faster and 6.6 times cheaper	Nate Herk	Not supported by the method
Same prompt, very different agent behavior	Nate Herk	Observed once, large enough to keep

What I believe after checking

I believe a harness can move results a lot for a fixed model on some tasks. ARC-AGI is real evidence for that, with caveats its own presenter supplied.

I do not think the general statement has been shown. That would need a decomposition, and the one researcher who tried reports parity elsewhere and a case where attribution was impossible.

My own focus on the harness is a bet about leverage. I can change a skill or a context file whenever I want. I cannot change the model.

What would move me toward the strong claim: the same model across several harnesses and several models inside one harness, at a fixed budget, repeated, on more than one benchmark, run by someone who owns none of the systems. A source for the 18 percent would help too. What would move me away: the ARC-AGI gap shrinking once the baseline gets the same borrowed prompt.

Key Takeaways

The best evidence in my notes is one same-model comparison on one benchmark, reported by the system's author with honest caveats.
The headline claims, an 18 percent gap and harnesses driving most recent gains, are asserted without a source.
A same-prompt product head-to-head changes harness and model together, so it cannot separate them.
Before trusting one, ask: was the model held constant, did it run more than once, are the costs real spend, and did the judge have a stake.

Sources

The cheap experiment I would run first is the one the head-to-head skipped: give both tools the same real task and vary only the prompt style.