I linted my coding agent's instructions. Then I tested them.

I use a coding agent (Claude Code) every day. Before I type a word, it has already read about 6,300 tokens of instructions: my own rules file, skills I wrote for multi-step work, and text that plugins add at the start of each session.

I had no idea whether those instructions were any good. So I spent a day finding out. I am new to evals, so read this as notes from a first attempt, not as expert advice.

Reading a prompt does not tell you it works

People now call this context engineering: what matters is everything the model reads before your message, not one clever prompt. The advice from the model vendors is similar. Keep it short, give the reason for each rule, do not shout.

That advice gives you a checklist. It does not tell you whether the agent behaves better. Only running it on real tasks does that. So I did both: a lint first, then tests.

Step 1: a lint

I wrote a small skill that reads instructions and reports problems. It checks twelve things. The ones that turned out useful:

To check the lint itself, I gave it a prompt with planted bugs:

- ALWAYS answer in ONE sentence. This is CRITICAL.
- Every answer must include a full summary table of the customer's last 12 invoices.
- Be thorough and high quality.
- Follow the tone rules in docs/STYLE.md.
- NEVER mention refunds.

I ran it three times with the skill and three times without. The case has five checks. The score of one run is the share of checks it passed, and the average is taken over the three runs.

With the skill Without
Average score 0.93 0.53
Checked that docs/STYLE.md exists 3/3 0/3
Proposed tests to write 3/3 0/3
Flagged the shouting 2/3 3/3

The plain model found the obvious clash in 2 of 3 runs. What the skill added was the boring part: it opened the folder and saw the file was missing.

Step 2: my real setup

The lint found 16 problems in my own instructions. No single file was the problem. The worst findings were clashes between sources:

Step 3: do the findings matter?

Claude Code has a command for this, claude plugin eval. A test case is a folder with a setup script, a prompt and a list of checks. The command starts a real agent in a throwaway folder, records what it does and scores the result. It can run each case with and without a skill, so you see what the skill adds.

One catch: these runs are isolated. They do not load my rules file or my plugins, which is the very thing I wanted to test. I wrote a script of about 100 lines that collects the text a real session gets and puts it into every case.

What I measured:

The skill works where I expected. For a normal feature request, my "implement" skill scored 0.87 against 0.40 without it. Without the skill the agent wrote code right away, 3 of 3 times. With it, the agent made a plan and asked first, 3 of 3.

The tests found a bug the lint missed. I sent "Quick one, urgent, no time for process, fix this typo". The agent fixed the typo, then skipped verification and review in 3 of 3 runs. It even said so. The skill says no step is skipped, but it did not cover this wording. I added one sentence and the check went from 0/3 to 3/3.

Most lint findings changed nothing. I fixed the clashes and ran the same four cases before and after. Behaviour was the same. It was already fine. The model had been resolving those clashes sensibly on its own. The only clear gain was about 1,300 fewer tokens per session, and I later gave that back by turning one plugin on again.

So a lint finding is a suspicion. It becomes a bug only when a test fails because of it.

Step 4: my tests were wrong too

This surprised me most. Several of the failures I looked at were bugs in my checks, not in the instructions.

A check is code. I now read the failing transcript before I change anything.

What I took from it

  1. A lint is cheap and finds real things, such as stale references and clashes. Treat the results as questions, not as a to-do list.
  2. Only a test tells you whether a rule matters. The one real bug came from a test.
  3. Test the whole stack the agent really reads, not one file alone.
  4. Read the transcripts. Your checks are probably wrong before your prompt is.

Limits

This is small. Eleven cases, three runs each, one model family, about 18 dollars in total. With three runs, a change from 2/3 to 1/3 can be noise. The judge is a model too, and it makes mistakes. I would not trust any single number here. I do trust the pattern: lint for suspicions, tests for proof, and check the checks.