How to know if your AI feature gives the right answers

You know an AI feature gives the right answers by testing it against a set of real questions with checked answers, and scoring every change against that set. That also lets you keep improving it: try a change, score it, and keep it only if the answers got better. Without a test set, a new prompt or a new model can quietly make the answers worse, and nobody notices until a customer does.

These test sets are called evals.

Why an AI feature needs its own tests

An AI model doesn’t give the same answer twice. A change to the prompt, the model or the documents it reads can fix one answer and break three others. Clicking through the app after each change doesn’t catch that.

A set of real questions with checked answers

The test set starts with real questions your users ask. Each one has an answer that someone who knows the business has checked. This is often called a golden set.

Every bug that reaches a user becomes a new question in the set, so the same mistake can’t come back unnoticed.

Scoring the answers

• Simple checks: the answer contains the facts it must, and nothing it mustn’t.
• Search checks, for features that search documents: the right passages were found.
• A second AI model as the judge: it compares each answer with the checked one and gives it a score. Spot-check the judge’s scores yourself now and then.

Improve it one change at a time

A test set lets you iterate. Try a new prompt, a different model or better documents, run the whole set, and compare the scores with the last run. Keep the change if the answers got better. Drop it if they got worse. Each round leaves the feature a little better, and you know by how much.

Keep the test runs honest

A score only counts if the run itself was clean. A test run on a machine that runs out of memory can look exactly like the AI getting worse. Runs with errors are thrown out, not compared.

Where a test set helps

Any AI feature whose answers matter to your business, for example:

• A support assistant that answers from your help articles.
• A tool that pulls details out of invoices, contracts or forms.
• A search over your company’s documents.
• An assistant that sorts or answers incoming emails.

FAQ

How many questions does a test set need? Start with the ones that matter most to your users, and add every bug you find. A small set that runs on every change beats a large one that never runs.

Does running the tests cost money? Each run calls the model once per question, so it costs a little each time. Keep the set focused enough to run on every change.

Can an AI feature be made 100% accurate? No. It can be measured, so you know how often it’s right and whether a change made it better or worse.