Why every internal AI assistant needs an evaluation set before launch

AI company brain dashboard where a hiring assistant answers with a cited source and shows a 36 of 40 evaluation score

Most teams test an internal AI assistant in a similar manner: a few individuals ask questions, the responses seem reasonable, and the assistant goes live. However, a few weeks later, someone receives a confident but incorrect response regarding a policy or number, leading to a collapse in trust. Consequently, the assistant is seldom used again.

The missing element is an AI assistant evaluation set. This is a fixed list of questions, typically around forty, with the correct answer for each documented before the assistant is developed.

What constitutes a good evaluation set

Real questions. Gather them from support tickets, chat channels, and email threads. If people genuinely ask it, it should be included on the list.

Known answers. Each question requires one agreed-upon correct answer, verified by the person responsible for that business area.

Questions it should decline. About a quarter of the list should consist of questions the assistant cannot answer based on its sources. A reliable assistant responds with "I don't know" and directs the question to a human instead of guessing.

A clear launch criterion. Determine in advance how many questions it must answer correctly, such as 36 out of 40, ensuring every unanswerable question is appropriately refused.

Why it changes the outcome

An evaluation set transforms "it seems fine" into a quantifiable metric visible to the entire team. It identifies real issues early, such as differing definitions of the same term by two departments or outdated documents. It also provides a method to verify that a new model or prompt adjustment hasn't inadvertently worsened performance.

Most importantly, it maintains honest scope. If you can't formulate forty clear questions and answers for a particular area, that area isn't ready for an AI assistant yet.

Start small

Select one focused area, draft its AI assistant evaluation set first, and then build only what is necessary to meet it. This approach is far more reliable than launching a company-wide assistant and hoping it functions correctly.

For the complete twelve-week approach, read this guide on how to build a company brain one focused area at a time.