I Built a Reviewer Skill From 67 Sourced Records, and the First Eval Failed

I use Jake Van Clief's Interpretable Context Methodology (ICM) to organize almost everything I build with agents. So I wanted a skill that could look at a project and tell me whether ICM fits it, fits part of it, or is not needed at all.

The easy version of that skill is a page of my own paraphrases with his name on top. That kind of skill drifts. After a while nobody can tell which lines are his position and which are a guess about his position, and that includes the model reading it.

So this one is built differently. Every rule in it traces to a record, every record carries a quote and a locator, and the skill file itself is generated. I never edit it by hand.

Records first, skill second

The unit is a record: one claim about when the methodology applies or what structure it requires, a verbatim quote that supports it, and a locator (a page in the paper, a file in the repository, a timestamp in a video or lesson).

There are 67 of them. 49 are boundary records, which say where the method does and does not apply. 18 are structure records, which say what a conforming workspace contains. The sources are his paper, his public repository, his videos and his free course material.

A script checks every quote against its locator. If the quote is not at that location in the captured source, the record fails validation and the build stops. This is the same idea as the timestamp anchors in my research notes: a claim that cannot be traced does not ship.

One caveat I want on the page. The records were mined and cross-checked by agents. No human has signed off on each one. The machine check proves the quote exists where the record says it does. It does not prove the record's reading of that quote is right.

The skill is a build artifact

The skill file is produced by a build step from the canonical records file. There are two outputs:

The split exists because quoting someone at length in a file I distribute is not mine to decide. The portable build keeps his claims attributable without carrying his words around.

Because the skill is generated, a drift check can compare the installed copy to the source and tell me when they disagree. Installing it also surfaced something I did not expect: my installed skill folder was a link into a public repository, so writing the private build there would have published it. The install now replaces the link with a real folder.

Freeze the answer key before you look

Before any records were mined, I wrote the eval: nine real cases. Two were clear fits, three clearly did not need the method, four were arguable. An answer key was drafted, hashed, and recorded with a timestamp.

Only then did the mining start.

The order matters. If the key is written after the skill, the key quietly bends toward whatever the skill already says. Pre-registration is cheap, and it is the only reason the next section is worth reading.

Run 1: 4 of 9

The first run matched the key on four of nine labels.

It did not fail randomly. It never called a clear fit "not needed", and it never called a clear not-needed case a fit. Both clear fits came back as "partial". Every miss sat on the line between fits and partial.

The cause was one sentence in the skill. I had defined partial as roughly "part of this is ordinary software". But the paper's own design puts scripts inside the workspace. By my definition, almost any real project was partial. The skill was following my wording faithfully, and my wording contradicted the source.

I fixed the definition and cited the record that decides it.

Run 2 is not a pass

The second run matched seven of nine, with all five clear cases right. One key entry changed too, because it rested on the same wrong definition. That revision was hashed and recorded before the rerun.

That number is a regression check and nothing more. The fix was written after seeing run 1, against the same nine cases, and the key moved with it. Of course it improved. Calling that a pass would be grading my own homework with the answers open.

So there was a third run on nine new cases, with the set and the skill frozen first. Clear cases: four of four. Arguable cases: four of five. The one strict miss was, again, fits versus partial.

After three runs the pattern was plain. The skill's judgment on the easy question (does this need the method at all) is solid. The definition of partial was still too loose. I narrowed it to one condition: a sourced no-go record applies to a named part of the project. Code and app views inside a workspace no longer count.

That narrowing is untested. The independent set was scored under the old definition, and I am not going to edit its key after the fact. Testing it needs a new pre-registered set.

Key Takeaways

• Build a methodology skill from records with quotes and locators, and generate the skill file. Hand-edited skills lose track of which lines are sourced.
• A machine check on quotes proves presence, not interpretation. Say which one you have.
• Write and hash the eval key before building the thing it grades.
• When a run fails, look at where the misses cluster. Mine were all on one boundary, which pointed at one sentence.
• A rerun after a fix on the same cases is a regression check. Only a fresh frozen set is evidence.

Sources

• Jake Van Clief, the ICM paper: arXiv 2603.16021
• Jake Van Clief's public ICM repository, videos and free course material, which the records quote and locate

Next is a new pre-registered set aimed only at the fits versus partial line. If the narrowed definition does not hold there, the definition is wrong, not the cases.