The problem
Every AI feature I've worked on had the same weak spot: nothing tells you when it gets worse. A reworded system prompt, a cheaper model, a provider update — none of them throw an exception. The bot just quietly stops following a policy, and you find out from a customer. Manual spot checks catch it for a week or two, then stop happening.
I wanted the AI equivalent of a unit test suite: write down what good answers look like once, and have CI refuse the pull request that breaks them.
The approach
- Suites in YAML, next to the code — each case is a prompt plus the checks its answer must pass. Cheap deterministic checks come first (contains, regex, length, valid JSON, JSON Schema); an LLM-as-judge check handles the rest, scoring a plain-language criterion from 1 to 5 with a second model at temperature 0.
- Sampling with pass rates — models aren't deterministic, so a case can be asked three times and pass on two good answers. One lucky answer proves little.
- Record once, replay in CI — a run against a live model can be recorded into a "cassette" file. CI replays it: deterministic, free, offline, and no API keys in the pipeline. Entries are keyed by a hash of the model, prompts and settings, so editing a prompt without re-recording fails loudly instead of passing on stale answers.
- Baselines, not perfection — results are compared with the last accepted run and every case is classified as passing, regressed, fixed, still failing or new. CI can fail only on regressions, so a known weakness doesn't block every unrelated change.
- Fits the .NET AI stack — it talks to any OpenAI-compatible endpoint
(Ollama, Groq, OpenAI, Azure OpenAI) and, from code, evaluates any
Microsoft.Extensions.AI
IChatClient. Output is JUnit XML, a Markdown job summary and JSON, and the repo ships a GitHub Action that runs its own example on every push.
The result
The example suite tests a fictional shop's support bot on local Ollama models, and the first real runs were more instructive than I expected. The 7B model passed 8 of 9 cases but kept writing an ocean poem when asked to, even after the system prompt explicitly banned creative writing. The 14B model passed all 9, at roughly 2.8× the runtime. Deleting one line from the system prompt made the 7B model tell customers that returns end at 30 days. Plumbline flagged it as a regression and failed the run.
That's the whole point: the trade-offs and breakages become visible in a pull request instead of in a support ticket. It's a small, focused v0.1 — single-turn prompts only for now — and it's open source under MIT.