Normal software fails loudly. A null reference throws, a test goes red, a dashboard spikes. AI features fail quietly. Someone rewords the system prompt to fix one complaint, and the bot stops following a refund policy it used to follow perfectly. Someone switches to a cheaper model to save money, and it starts helping people with things it was supposed to decline. No exception, no alert — just a slow leak of wrong answers that you hear about from a customer.
In my lessons from shipping an internal agent, building the evaluation set before the agent was lesson four. This post is the practical version: the small tool I built for it, called Plumbline, and what it caught the first time I ran it.
Write down what "good" means, as tests
The core idea is boring on purpose. Every AI feature gets a test suite: a list of real prompts, each with the checks its answer must pass. In Plumbline that's a YAML file next to the code:
- id: refund-store-credit-window
prompt: My hiking boots arrived 45 days ago and they're too small. Can I return them for cash?
assert:
- regex: (?i)store\s+credit
- judge: Does not offer a cash refund, and offers store credit instead.
Two kinds of checks do almost all the work:
- Cheap, exact checks first. Does the answer contain the right price? Does it avoid a forbidden phrase? Is the triage output valid JSON that matches the schema the next system expects? These are fast, free and never ambiguous. Use them wherever you can.
- A judge for the rest. Tone, "stays on topic" and "doesn't promise anything outside the policy" don't fit in a regex. For those, a second model scores the answer from 1 to 5 against a plain-language criterion, at temperature 0. It's not perfect — a judge is only as good as the judge model — so I check each criterion against a few hand-graded answers before trusting it.
Models aren't deterministic, so a case can also be asked several times with a required pass rate: "two out of three answers must be friendly" is a more honest test than one lucky sample.
Make it cheap enough to run on every pull request
A test suite nobody runs is decoration. Two things make this one run on every change:
Record once, replay in CI. Running against a live model costs money, needs API keys in the pipeline, and gives slightly different answers each time. So Plumbline records a live run into a "cassette" file that gets committed. CI replays it — offline, free, in about a tenth of a second. Each recording is keyed by a hash of the model, the prompts and the settings, so if you change a prompt and forget to re-record, the run fails loudly instead of quietly passing on old answers.
Compare against a baseline, not against perfection. Every model has weaknesses you've decided to live with. If the suite demands 100%, it's red forever and people learn to ignore it. Instead, each run is compared with the last accepted run, and every case is classified: still passing, regressed, fixed, still failing, or new. CI blocks the pull request only for regressions and new failures. Known weaknesses stay visible in the report without blocking unrelated work.
What it found on day one
I wrote an example suite for a fictional outdoor-gear shop's support bot — refund windows, shipping costs, off-topic requests, a prompt-injection attempt, and a ticket-triage step that must produce valid JSON — and ran it on two local models through Ollama.
- The 7B model passed 8 of 9. It handled the refund rules, the shipping threshold, the triage JSON and the injection attempt. But asked for a poem about the ocean, it wrote the poem. I made the system prompt explicitly ban creative writing. It wrote the poem anyway, just with an offer to help with orders in front of it. Prompt wording alone didn't fix it.
- The 14B model passed all 9, but took about 2.8 times as long on the same machine. That's the real decision a team faces, and now it's a number in a report instead of a hunch.
- Then I broke it on purpose. I deleted the one line of the system prompt that describes store credit for returns between day 31 and day 90. The 7B model immediately started telling customers that returns end at 30 days — wrong, and the kind of thing that turns into a complaint. Plumbline marked the case as regressed, the judge scored the answer 1 out of 5, and the run failed. In a real project, that pull request doesn't merge.
None of these would have shown up in unit tests, and the first one would have taken a long time to surface in production.
If you take one thing from this
You don't need my tool to do this — you need the habit. Collect twenty real questions your AI feature must handle, write down what a good answer has to contain, and run them whenever the prompt, the model or the provider changes. Start with exact checks, add a judge only where you must, and fail the build on regressions rather than on imperfection.
Plumbline is open source on GitHub if you want a head start: .NET 10, works with Ollama, Groq, OpenAI and Azure OpenAI, plugs into Microsoft.Extensions.AI, and ships a GitHub Action.
Shipping an AI feature and not sure how you'd know if it got worse? Tell me what you're building — I'll suggest the first ten tests I'd write for it.