Classic tests check whether a function returns exactly the expected output for an input. With language models that rarely works: a good summary can be phrased many ways, and the same request never returns the same text twice. The quality of AI features can still be measured. You just have to stop looking for the one right answer and check instead whether an answer has the properties that matter.
The evaluation set
Every evaluation rests on a collection of typical inputs for which you know what a good answer looks like. The best sources are real user requests, anonymised and selected for frequency and difficulty. Deliberately include hard cases too: ambiguous questions, inputs with typos or dialect, requests the feature should decline. Fifty to a hundred well-chosen examples are worth more at the start than a thousand random ones.
For each example, record what makes a good answer: which facts it must contain, which source it should cite, which format is expected, what must not appear. These criteria are the real work — and they force the team to agree on what the feature is supposed to do.
Criteria you can check
Some properties can be checked automatically and unambiguously. Is the output valid JSON matching the schema? Does it stay under the maximum length? Does it contain the required fields, a source, no email addresses? Such deterministic checks are fast, cheap and reliable and should form the first part of every evaluation.
Other properties need judgement: is the answer correct, complete, in the right tone, supported by the sources? There are two complementary routes: human review and evaluation by a second model against clear criteria.
Models as judges — with checks
A language model can judge an answer against a precise rubric, for example: «Does the answer contain all three stated facts? Does it claim anything not in the sources?» That scales and makes frequent evaluations affordable. But models as judges have their own weaknesses: they favour longer answers, are swayed by confident wording and are unreliable on subtle domain errors. So use single, clearly defined criteria instead of an overall score, and regularly have people review a sample to see whether the judge model agrees with human judgement.
Regression tests on every change
The biggest value of an evaluation lies in comparison. Every change to the prompt, model, retrieval or post-processing is tested against the same set. This shows whether an improvement in one case makes something worse elsewhere — which happens with prompt changes more often than expected. A model switch, whether to a new version from the same provider or to another provider, should never happen without this comparison.
Because model outputs vary, run each example several times for important decisions. A change that looks better in a single run can be worse on average.
Evaluation in production
The set only covers what you anticipated. In production, new requests, documents and user behaviour appear. User feedback, samples of real requests and automatic checks for known failure patterns complement offline evaluation. Cases where the feature failed go into the set, so the same error does not return after the next change.
Typical mistakes
Most often we see teams optimising prompts by impression: a few test questions in the playground, the answer looks good, the change ships. Add sets containing only easy cases, overall scores from judge models without human checks, and no repetitions despite varying results. All lead to the same outcome: the team believes it knows the quality, and does not.
A lean start
- Collect fifty real, anonymised requests, including hard cases.
- For each, record what a good answer must and must not contain.
- Automate deterministic checks for format, length and forbidden content.
- Use a judge model for content criteria and calibrate it against human samples.
- Run every prompt or model change against the set.
Innopulse builds AI features with such an evaluation set from the start, because it is the basis for every later improvement and every model switch.