The Order to Build a Skill In

A skill is a markdown file with a name, a description and a body of instructions. That makes it tempting to write one like a README: think about the workflow, type out the steps, install it.

I maintain a library of dozens of skills for Claude Code, kept as files in my vault and synced to the live skills folder. When I went through my research notes on what separates the skills that hold up from the ones that do not, the same answer kept coming back. The order matters. Do the task by hand, write down what you did, turn that into a skill, then test it against cases. Most broken skills skipped a step.

Step one: do the task by hand

Ras Mic, on Greg Isenberg's channel, gives the clearest version. He crosses out the usual sequence (identify a workflow, write the skill) and replaces it with four steps: pick a specific workflow, walk through it with the agent in conversation, verify a successful run, and only then ask the agent to write the skill from what it just did.

"By hand" here means in the conversation, one step at a time, correcting as you go. His example is a skill that screens sponsor emails. The first attempt came back positive on every sponsor, because nothing in it said what a rejection looks like. The rejection criteria only appeared once he worked through real emails and corrected the agent.

That example carries the argument. Instructions written before a run are guesses about what the task needs, and a model handed guesses returns something plausible.

An NVIDIA team describes the same habit for its video analytics skills: go through the workflow with an agent, then ask for the skill afterwards. Neither source compares a skill written cold with one built from a run. This is an argument from mechanism and practice. I find it convincing, but I would not call it measured.

Step two: write down what you did

Ras Mic folds this into the next step by asking the agent to review the session and write the file. I would keep it separate. Before asking for a skill, list what the run needed: the steps in the order they happened, every correction you made, and what a bad result looks like.

A successful conversation contains a lot that should not survive into the skill, and Nick Nisi's talk is the warning. He generated more than 10,000 lines of skills from his company's docs, and his evals showed they were hurting: one skill scored 77 percent correct when loaded and 97 percent without it. He replaced the lot with 553 hand-written lines about the things models reliably get wrong, and reports eval runs dropping from 68 minutes to 6. His line is that "the models know how to code" and only need to be told where they trip.

One caution: that is one skill on one product's evals, from a sponsored conference slot. The direction still matches step one. The corrections you made by hand are the valuable part, and the write-down is where you separate them from everything else.

Step three: turn it into a skill

By now the file is mostly transcription, with two decisions left.

The first is the description. Only a skill's name and description sit in context until it is invoked. In Ras Mic's demo that was 53 tokens for the index entry against 944 for the body. So the description is the trigger, and it needs the words a real request would contain. Matt Pocock adds a choice I had not framed before: a skill the model can invoke on its own keeps its description in context permanently, while a skill only I invoke costs nothing there but leaves me to remember it.

The second is size. Pocock's test is to delete a paragraph and see whether behavior changes. If it does not, the paragraph was never an instruction.

Step four: test it against cases

This is the step my own library is missing. As of late June, none of my skills had an eval behind it, and I have a concrete reason to care. During one research run, a stale cache in my frame-extraction tool silently served another video's frames. Every step of the skill ran. The frames were still the wrong ones.

Chase AI's walkthrough of Anthropic's updated skill-creator plugin supplies the vocabulary. A capability-uplift skill makes the model better at something it is weak at, and the test is a blind comparison with and without the skill. An encoded-preference skill captures a workflow the model can already do, in my order and style, and the test is fidelity: did every step run, in sequence.

I would not lean on his headline benchmark, though. The table on screen shows a PDF skill passing 5 of 5 cases with the skill and 2 of 5 without. Five cases is a smoke test, and the video discusses no downsides. I take the mechanics and leave the number.

For how many cases to start with, John Groetzinger of Cisco suggests three to five evals as the bare minimum for one skill. And when a case fails, Ras Mic's loop applies: ask the agent why, fix it in the conversation, then have it update the skill file so the fix persists. He reports five such loops before his analytics skill ran unattended, though he does not show the loops themselves.

Which step was skipped

This mapping is mine, assembled from the sources above.

Symptom	Step that was skipped
Output is plausible but never says no	One: no real run produced the criteria
Skill is long and the agent ignores parts of it	Two: the whole transcript or doc was encoded
Skill never fires, or fires on the wrong request	Three: the description was an afterthought
It worked last month and nobody noticed it stopped	Four: no cases to rerun

Key Takeaways

A skill encodes a successful run. Written before that run exists, it encodes guesses.
The useful content is the corrections you made by hand. Write those down before generating the file, and leave the rest out.
The description is the trigger, and it sits in context whenever the model can invoke the skill itself. Decide that on purpose.
A skill without cases can break silently. Three to five cases is a reasonable floor, and a failed case is fixed in the skill file.

Sources

Next for me is step four: a fidelity eval for my research skill, starting with the stale-cache case that already got past me once.