How do you evaluate an LLM application before it goes live?
Agree the pass mark before testing, build the evaluation set from real user inputs including the awkward ones, report results for every user group rather than one average, and re-run the same versioned tests on every release. Retrieval helps but does not remove errors — published evaluations of commercial retrieval tools still found 17–33% hallucination rates.
A demo shows an application at its best. Evaluation exists to show it at its worst, before users find that out for themselves. Four practices separate real evaluation from a well-rehearsed demo.
Agree the pass mark before you test
Acceptance criteria written after the results always move. Decide up front what the system must achieve — what counts as a correct answer, what it must never do, and how much wrongness is tolerable in which situations — and have the business owner sign it. A result can then be judged, not negotiated.
Build the evaluation set from real inputs
Collect real questions and documents from the people who will use the system, including the messy ones: ambiguous phrasing, missing context, questions with no answer in the source material. Retrieval grounding reduces errors but does not remove them — a preregistered study published in the Journal of Empirical Legal Studies in 2025 found that commercial retrieval-based legal research tools still produced hallucinations between 17% and 33% of the time. The interesting cases are exactly the ones a demo avoids.
Test the population, not the average
One overall score hides who the system fails. Performance should be reported separately for every group the system serves — user types, document types, languages, edge conditions. And a high score on a held-out test set is not the end of it: work published in the Journal of Machine Learning Research in 2022 showed that training pipelines routinely produce models that score identically in testing yet behave very differently once deployed. Stress tests — out-of-scope questions, adversarial phrasing, drifted inputs — belong in the set from the start.
Make the evaluation permanent
An evaluation that runs once is a snapshot; the system will change the week after. Version the evaluation set, wire it into the release process so it can stop a deployment, and re-run it on every change to the model, the prompts or the retrieval index. Findings should trace to tests, and tests to requirements — that traceability is what turns an evaluation into evidence a reviewer, a buyer or a regulator can actually check.
Related questions
Does retrieval-augmented generation stop hallucinations?
It reduces them, it does not stop them. A 2025 Journal of Empirical Legal Studies evaluation found commercial retrieval-based legal research tools hallucinating between 17% and 33% of the time. Plan for evaluation and human review where errors are costly.
Why is one overall accuracy score not enough?
Because equal test scores can hide unequal behaviour. JMLR 2022 research showed pipelines producing models that score the same on held-out data yet diverge in deployment, and averages routinely conceal failures on specific groups.
How often should the evaluation run?
On every change — model, prompt, or retrieval index. The set is versioned so results are comparable, and it runs as a release gate that can stop a deployment, not as a report read afterwards.
AI Testing & Certification
Test AI solutions for accuracy, safety and conformity, and prepare them for external certification.
Discuss your requirement