Evals in CI: testing what is not deterministic

A team that puts an LLM into production discovers an uncomfortable problem early: the test suite that guaranteed quality for years does not work on the new feature. Equality assertions fail when the correct output can be written ten different ways.

Why deterministic tests do not cover this

Traditional tests compare output against an expected value. LLM output varies between runs, is sensitive to tiny prompt changes, and can be right using words completely different from the ones in your fixture.

The common reflex is to give up on testing and validate by hand before each release. That works until the third prompt change, when nobody remembers which behaviour was intentional and which was a regression.

The way out is not abandoning tests, it is changing the question. Instead of "is the answer exactly this?", ask "does the answer satisfy these properties?".

Three layers, cheap to expensive

Layer 1, deterministic assertions on the output. More fits here than people expect: the JSON is valid and matches the schema, the answer cites at least one source, it stays under N characters, it contains no PII, the tool call used the right parameters. Fast, stable, and it catches most integration regressions.

Layer 2, a golden set with programmatic checks. Thirty to a hundred real cases with verifiable properties: contains the correct figure, lands in the right category, refuses when it should refuse. Runs in seconds and detects most quality breaks.

Layer 3, LLM as judge. For what is left: tone, completeness, usefulness. It is the most expensive and least reliable layer, which is exactly why it should be the smallest.

The judge has known biases

Using one model to evaluate another works better than intuition suggests. The work that popularised the practice measured over 80% agreement between strong LLM judges and human preference, comparable to agreement between two humans.

That same work documented the biases, and ignoring them invalidates the result:

• Position. The judge tends to prefer whichever answer came first. Mitigation: swap the order and run twice.
• Verbosity. Longer answers score better even without added substance. Mitigation: an explicit criterion penalising redundancy.
• Self-preference. Models tend to favour text in their own style. Mitigation: judge with a different model family than the one that generated.

A judge without an explicit rubric measures style. With a rubric of three to five objective criteria and a short scale, it becomes a usable signal.

What runs in CI and what runs outside it

The temptation is to run everything on every push. Do not: model calls cost money and time, and a ten-minute suite stops being run in practice.

A split that works: layers 1 and 2 on every pull request, with a small model at temperature zero, targeting under two minutes. Layer 3 on merge to the main branch and before release, on a sample. The result becomes a versioned artifact so you can compare releases.

Treat variance as signal, not noise: run the critical cases a few times and record the spread. A feature swinging between 70% and 95% on the same set is not ready, however good the average looks.

Holistic evaluation, not a single score

Efforts like HELM consolidated an idea that applies to products too: evaluate across multiple dimensions and scenarios rather than collapsing everything into one number. A high average accuracy can hide systematic failure in a category that is 5% of volume and 80% of risk.

In practice, break the set down by usage scenario and look at the worst one, not the mean. A support assistant that answers billing questions well and refund questions badly does not have a quality problem on average, it has a quality problem exactly where the company is exposed.

What this means for your team

Start today with layer 1: schema, format, presence of a citation, absence of PII. It needs no framework and already blocks the most common class of regression.

Build the golden set from real traffic, not invented cases. Every time a bug reaches production, the case joins the set. In three months you have a ruler that reflects your problem instead of a generic benchmark.

Only then add the judge, with a written rubric and alternated ordering. And version the results: without history, you cannot say whether swapping models improved or degraded your application, which is the question you will be asked the week a new model ships.