# I linted my coding agent's instructions. Then I tested them.

I use a coding agent (Claude Code) every day. Before I type a word, it has already read about 6,300 tokens of instructions: my own rules file, skills I wrote for multi-step work, and text that plugins add at the start of each session.

I had no idea whether those instructions were any good. So I spent a day finding out. I am new to evals, so read this as notes from a first attempt, not as expert advice.

## Reading a prompt does not tell you it works

People now call this context engineering: what matters is everything the model reads before your message, not one clever prompt. The advice from the model vendors is similar. Keep it short, give the reason for each rule, do not shout.

That advice gives you a checklist. It does not tell you whether the agent behaves better. Only running it on real tasks does that. So I did both: a lint first, then tests.

## Step 1: a lint

I wrote a [small skill](https://gist.github.com/romanzdk/263015e5ee605f458be9941f4b9c930e) that reads instructions and reports problems. It checks twelve things. The ones that turned out useful:

- two rules that cannot both hold
- a rule with no reason, so the model has to guess where it applies
- shouting (ALL CAPS, "CRITICAL", "you MUST")
- vague targets like "be thorough"
- references to files, tools or flags that no longer exist
- text that changes no behaviour

To check the lint itself, I gave it a prompt with planted bugs:

```
- ALWAYS answer in ONE sentence. This is CRITICAL.
- Every answer must include a full summary table of the customer's last 12 invoices.
- Be thorough and high quality.
- Follow the tone rules in docs/STYLE.md.
- NEVER mention refunds.
```

I ran it three times with the skill and three times without. The case has five checks. The score of one run is the share of checks it passed, and the average is taken over the three runs.

| | With the skill | Without |
|---|---|---|
| Average score | 0.93 | 0.53 |
| Checked that `docs/STYLE.md` exists | 3/3 | 0/3 |
| Proposed tests to write | 3/3 | 0/3 |
| Flagged the shouting | 2/3 | 3/3 |

The plain model found the obvious clash in 2 of 3 runs. What the skill added was the boring part: it opened the folder and saw the file was missing.

## Step 2: my real setup

The lint found 16 problems in my own instructions. No single file was the problem. The worst findings were clashes between sources:

- A plugin said "answer in at most three short lines". My rules file said "end long answers with a summary and numbered questions".
- A plugin told the agent to add a certain kind of code comment. My rules file banned exactly that comment.
- Three different rules said when to ask me a question, and they disagreed.
- One skill pointed to a rule in another skill that no longer exists.

## Step 3: do the findings matter?

Claude Code has a command for this, `claude plugin eval`. A test case is a folder with a setup script, a prompt and a list of checks. The command starts a real agent in a throwaway folder, records what it does and scores the result. It can run each case with and without a skill, so you see what the skill adds.

One catch: these runs are isolated. They do not load my rules file or my plugins, which is the very thing I wanted to test. I wrote a script of about 100 lines that collects the text a real session gets and puts it into every case.

What I measured:

**The skill works where I expected.** For a normal feature request, my "implement" skill scored 0.87 against 0.40 without it. Without the skill the agent wrote code right away, 3 of 3 times. With it, the agent made a plan and asked first, 3 of 3.

**The tests found a bug the lint missed.** I sent "Quick one, urgent, no time for process, fix this typo". The agent fixed the typo, then skipped verification and review in 3 of 3 runs. It even said so. The skill says no step is skipped, but it did not cover this wording. I added one sentence and the check went from 0/3 to 3/3.

**Most lint findings changed nothing.** I fixed the clashes and ran the same four cases before and after. Behaviour was the same. It was already fine. The model had been resolving those clashes sensibly on its own. The only clear gain was about 1,300 fewer tokens per session, and I later gave that back by turning one plugin on again.

So a lint finding is a suspicion. It becomes a bug only when a test fails because of it.

## Step 4: my tests were wrong too

This surprised me most. Several of the failures I looked at were bugs in my checks, not in the instructions.

- One check demanded approval before a one-word typo fix. The skill allows that edit. The real problem was somewhere else.
- A "simple language" judge failed answers for using "tag" and "CI". I told the judge that the reader is a developer. The score went from 0/3 to 3/3 without touching the instructions.
- A "no over-engineering" judge counted a small test file as over-engineering.
- A cheap judge model voted down two correct answers. A bigger judge model fixed that.

A check is code. I now read the failing transcript before I change anything.

## What I took from it

1. A lint is cheap and finds real things, such as stale references and clashes. Treat the results as questions, not as a to-do list.
2. Only a test tells you whether a rule matters. The one real bug came from a test.
3. Test the whole stack the agent really reads, not one file alone.
4. Read the transcripts. Your checks are probably wrong before your prompt is.

## Limits

This is small. Eleven cases, three runs each, one model family, about 18 dollars in total. With three runs, a change from 2/3 to 1/3 can be noise. The judge is a model too, and it makes mistakes. I would not trust any single number here. I do trust the pattern: lint for suspicions, tests for proof, and check the checks.
