LLM evaluation means systematically checking a language model’s outputs against expectations defined in advance. It answers the question of whether a change to the system — a different prompt, a different model, a different retrieval strategy — actually improves the result or merely makes it different.
Why classic tests do not work
A conventional software test compares an output with an expected value. With language models that fails, because the same task has many equally correct answers differing in wording. An exact comparison therefore fails on correct answers and is unusable. Checking must instead attach to properties: does the answer contain the required information? Does it hold the format? Does it name a source? Does it avoid a particular statement?
The test set is the actual tool
At the core of every evaluation is a collection of inputs with the expected properties of the output. That collection need not be large, but it must contain the difficult cases: edge cases, ambiguous queries, inputs outside the intended scope, and the cases that went wrong in the past. A test set covering only the normal case merely confirms that the system works when nothing difficult happens.
Kinds of checking
Three kinds complement each other. Rule-based checks verify what is objectively determinable — format, presence of specific details, adherence to limits — and are cheap and reliable. Model-based scoring has one model judge another’s output against given criteria; that scales well but carries its own biases and itself needs calibrating against human judgement. And human review remains necessary for anything requiring professional judgement — it is expensive and therefore to be deployed deliberately.
Regression is the real purpose
The greatest practical value of evaluation lies not in measuring absolute quality but in detecting degradation. With language models that matters particularly, because a prompt change improving one case can degrade several others unnoticed. Without a test set, every change is a bet. With one, it is a measurement.
Limits of automatic scoring
Automatic scoring measures well what can be clearly defined and poorly what requires judgement. Whether an answer is professionally appropriate, correct in tone, or comprehensible to the specific audience largely escapes a rule. Ignoring that limit means optimising the system for the measurable quantity while degrading exactly what is not measured.
Practical consequence
A test set of a few dozen well-chosen cases costs a day of work and fundamentally changes how an AI system is worked on: discussions about impressions become comparisons of results. That investment pays off earlier than most teams assume — sensibly before the first major prompt revision, not after it.