Every morning I get a short digest of practical AI news for developers, in Czech. A small Python project makes it. I built it with a coding agent, and the code is public.
This post is about one design question in it, and how I answered it with evals instead of a guess. I am still new to evals, so read it as notes, not as expert advice.
Why not just RSS and a summary
A plain summary of feeds has four problems:
- it repeats one event from five sites
- it sells an old project as new, because somebody posted it again
- it turns "announced" into "available"
- it drops the small print: paid plan only, preview, one region
So the real work is choosing, checking and dating. Summarising is the easy part.
My first version was a general-purpose agent with three sub-agents. It worked, but one digest used about 11 million tokens, most of them cached. At list price that is about $2.71 a day. The version below costs 2 to 3 cents. That comparison is rough: one day, and not exactly the same sources.
The design
Plain code does the routine work. A model is used in two places only.
- Collect (code). Read about a dozen public sources for one calendar day.
- Triage (model). Group links about the same event and drop the noise.
- Verify (model). Read the pages of each event and decide: what changed, when, stable or preview, who can use it.
- Select (code). Drop what is not new or not confirmed. Keep the top seven.
- Report (code). Write the digest and save every page that was read.
The flow is a straight line with one loop, so there is no agent framework. It uses Pydantic AI for the two model steps, because the output is a typed object and a rejected answer can be sent back to the model.
One rule is enforced in code, not in the prompt: every URL an item cites must be a page the model really read.
The question
In the verify step, the model may read up to four more pages when a key fact is missing. For example, the announcement says "now available" and the limits are on a linked pricing page.
That is the only "agentic" part. It makes the step slower and less predictable. Is it worth it?
To find out, I added a switch. The single-pass version gets the same pages but cannot open new ones. Everything else is the same.
The test
- 20 hand-written cases: 15 for verify, 5 for triage.
- All products are made up, so the model cannot answer from memory.
- The cases cover the four problems above, plus a quiet day, an unreachable source and a page with a hidden instruction for AI systems.
- Split into dev cases, used while tuning the prompts, and holdout cases for the final check.
- Each case runs three times.
- Most checks are plain code: include or not, kind of change, availability, date. Two model judges check that the item keeps the key condition and that every statement is supported by the pages.
Results are stored in Braintrust, so I can compare two runs case by case.
What came out
Holdout cases, Claude Haiku 5.5:
| Reads more pages | Single pass | |
|---|---|---|
| Right include / exclude decision | 1.00 | 1.00 |
| Availability (stable, beta, announced) | 1.00 | 0.92 |
| Key condition kept in the item | 0.92 | 0.70 |
| Model requests per event | 1.29 | 1.00 |
| Cost per event | $0.0009 | $0.0007 |
Three things I read from it:
It helps in one kind of case. The model used the extra reading in about a third of the cases. Only those improved: the ones where the condition (beta, waitlist, quota) sits on a linked page.
It does not change what gets published. The include or exclude decision was the same in both versions. Extra reading makes an item more exact. It does not find different news.
It costs about 30% more per event. At under a tenth of a cent per event, I pay that.
I also replayed one real day with both versions. Both gave four items with the same main stories. The reading version made 14 model requests ($0.020), the single-pass one 8 ($0.015).
The bigger win was not the agent
The main quality problem was small invented details. An item said a tool was "open-source" when no page said so.
I added a prompt rule against it. It still happened in about one of ten published items.
What fixed it was making the items shorter. Each field got a word limit of 8 to 15 words. The "every statement is supported" score on holdout went from 0.91 to 1.00, and from 0.73 to 1.00 for single pass. Fewer words leave less room to invent.
It had a price. In one trial of twelve, a short item kept one limit of a free plan and dropped the second.
Limits
- The holdout is not clean. It ran five times while I was still changing prompts, and one rule came from a holdout failure. The numbers are probably a bit optimistic.
- The expected answers were written by the coding agent. I have not reviewed all of them yet.
- The judge is the same model as the writer.
- It is small. 20 cases, three runs each. A score moves when one case flips.
- The test code had bugs too. One check looked for forbidden text in items that were never published.
What I took from it
- Add the baseline switch early. Without the single-pass version I would only have a feeling that the agent part helps.
- Look at which cases improve, not only at the average. Here it was one kind of case.
- Try the boring fix first. A word limit did more for quality than the extra reading.
- Write down how clean the final check really was.