AI Chatbot Testing Is the Part Everyone Skips

Your support bot answered that billing question correctly last Tuesday. Today it gets it wrong. Nobody on your team touched the prompt, the help docs, or a single line of code.

Something underneath you moved. That is the failure mode nobody plans for, and AI chatbot testing is the only thing that catches it before a customer does.

Here is the uncomfortable part. Almost every team we talk to has a deploy pipeline for their app and nothing at all for their agent. They ship a prompt change the way you'd ship a tweet — write it, eyeball two replies in a staging widget, push. Then they wonder why the thing that demoed beautifully in March is quietly making up refund policies in September.

The thing that breaks is rarely your code

Build on a hosted model and you have accepted a dependency you can't pin the way you pin a library version. Every major provider — OpenAI, Anthropic, AWS Bedrock — now publishes a deprecation calendar, and every retirement resets the behaviour of anything wired to that version (Lyceum Technology's write-up on version pinning is a good primer). Providers can also update weights behind an endpoint without a breaking API change and without a version flag in the response.

The symptoms are boring and nasty. Tool-call formatting shifts. JSON adherence loosens by a hair. Refusal boundaries move, so a question the bot used to answer now gets a polite deflection. The model gets more verbose and your token budget stops fitting the retrieved context, so the third document — the one with the actual answer — silently falls out of the prompt.

None of that throws an exception. Your error rate stays at zero. Your logs look perfect. The only signal is that customers are slightly angrier this month, and by the time that shows up in CSAT you've lost the ability to say which change caused it.

Model drift isn't the only trigger, either. Someone rewrites a help article. Someone adds forty new docs and the retriever's ranking shifts. Someone tightens the system prompt to fix one weird answer and breaks eleven normal ones. Each of those is a deploy, and none of them gets tested.

What an AI chatbot testing suite actually contains

Forget the tooling for a second. The asset is the test cases, and you can build a useful set in an afternoon with a spreadsheet and a text editor.

The golden set

Fifty to a hundred real questions with known-good answers. Pull them from your actual inbox, not from your imagination — the phrasing customers use is worse than the phrasing you'd invent, and that's the point. "why was i charged twice in july" belongs in the set. "Could you please explain the duplicate billing policy?" does not.

For each one, write down what a correct answer must contain: the fact, and the source document it should cite. If you're running a grounded bot, the citation is half the test. An answer that's right but cites the wrong page will be wrong next month when that page changes.

The refusal set

The questions your bot should not answer. Pricing exceptions. Legal commitments. Anything involving a discount it isn't authorised to offer. Anything about another customer's account.

This set matters more than the golden set and almost nobody builds it. In April 2025 Cursor's support bot invented a policy limiting the product to one device per user; customers read it as official and started cancelling. The bot wasn't broken in any way a smoke test would find. It answered confidently, in the right tone, about a policy that did not exist. We wrote about why support bots invent answers and what grounding actually fixes — but grounding is a design choice, and the refusal set is how you verify the design still holds after today's deploy.

The multi-turn cases

Real support conversations are not one question and one answer. The customer asks something, gives a partial detail, corrects themselves, then asks a follow-up that only makes sense given turn two. A bot that scores brilliantly on single-turn questions can still lose the thread on the fourth message and start answering a question nobody asked.

You need a handful of these written as scripted conversations, with an expected outcome at the end — including the ones where the right outcome is a handover to a person. If your escalation path only gets tested by accident in production, it will fail in production.

"It sounded right" is not a test

The tempting shortcut is to check the final answer and move on. It reads well, it's factually correct, tick.

That misses most of what goes wrong. An agent that retrieved the wrong document and then guessed its way to a correct-sounding answer has passed your test and is one question away from failing badly. Adaline's 2026 evaluation guide puts the gap between final-output scoring and full trajectory scoring at 20–40% more test cases passed when you only look at the end — a vendor figure, so treat the exact range with some salt, but the direction matches everything we've seen. Behaviour that changes at step three corrupts steps four through eight, and single-turn scoring can't see any of it.

So log the trace, not just the reply. Which chunks came back from retrieval, in what order, with what scores. Which tools were called and with what arguments. What actually landed in the prompt after truncation. When a test fails, that trace tells you whether you have a retrieval problem, a prompt problem or a model problem — three completely different fixes that look identical from the outside.

This is also where self-hosting earns its keep. When the whole stack runs on your infrastructure, the trace is just data in your own database, and you can replay a month of real conversations against a candidate model without asking anyone's permission or paying per replay.

How much should you trust an LLM to grade your bot?

Grading a hundred answers by hand every deploy is not going to happen, so most teams end up with a model marking another model's homework. It works better than it sounds. It also has a ceiling that's worth knowing about before you build your gates on top of it.

A 2025 study in global health compared LLM judges with human raters and found something awkward: AI–AI pairs agreed with each other more than humans agreed with humans (mean κ = 0.53 versus 0.32), but AI–human agreement sat below both at κ = 0.22 (medRxiv). Consistent, in other words, without necessarily being consistent with you.

Clinical safety work using a jury of models rather than a single judge did better — κ = 0.74 against human consensus — but agreement on individual criteria ranged from 0.34 to 0.97 (arXiv). That spread is the practical lesson. Judges are reliable on concrete, checkable criteria and unreliable on vague ones. "Does this answer contain the refund window stated in doc #412?" is a question a model can grade. "Is this answer helpful?" is not.

Write your rubrics as binary, factual checks and the judge becomes genuinely useful. Write them as vibes and you've automated a coin flip.

Every production failure becomes a test case

The best eval sets aren't designed. They accumulate.

Someone complains that the bot gave a wrong shipping estimate. You fix it. Then — and this is the step that gets skipped — you add that exact question to the suite with the correct answer attached, so the fix can never silently un-fix itself. Do that for six months and you have a test set shaped like your actual customers rather than like your assumptions about them.

Two habits make this stick. Give agents a one-click way to flag a bad answer from the inbox, so capturing a case costs nobody anything. And review the flagged pile weekly, because half of them won't be bot failures at all — they'll be a help doc that's wrong, or missing, and no amount of AI chatbot testing fixes a knowledge base that doesn't contain the answer. We've written separately about getting help centre content in shape first.

Gate the deploy, or don't bother

A test suite you run when you remember is a document, not a gate.

Wire it to run on every change to the prompt, the model version, the retrieval config and the knowledge base. Set a pass threshold — start loose, tighten it as the set matures — and fail the deploy underneath it. Pin your model version explicitly and treat the provider's deprecation date as a scheduled piece of work, with a full suite run against the replacement before you switch.

Then keep a smaller version running against live traffic. A handful of binary checks on real conversations — did it cite a source, did it stay inside the refusal set, did it hand over when it should have — will surface a silent weight update within hours rather than within a quarter.

None of this is exotic. It's the same discipline any team applies to a payments integration. The reason it feels optional for support bots is that the failures are quiet: a wrong answer looks exactly like a right one until somebody checks.

What this has to do with measuring anything else

There's a connection here to the metric everyone reports. If you don't know your bot is answering correctly, your containment rate is measuring how often customers gave up, not how often they got helped — which is why the same product can honestly claim 76% and 14% depending on what it counts. An eval suite is what makes a deflection number mean something, because it's the only evidence that the conversations you contained were actually resolved.

The World Quality Report 2025–26 found 89% of organisations piloting or deploying generative AI in quality engineering, but only 15% at enterprise scale, with half saying they lack the in-house expertise (summarised by Applause). The gap between piloting and scaling is mostly this: nobody can prove the thing works well enough to expand it.

Start with fifty questions. Write down the right answers. Run them before every deploy. That's a smaller project than the tooling market wants you to believe, and it's the difference between a support agent you can expand and one you quietly turn off in six months.

Written by the ChatterMate team — we build an open-source AI support agent, and we run our own answers against a golden set before we ship.

ChatterMate is open source and free to start, with the first 300 chats free and full self-hosting if you'd rather keep every trace on your own hardware. Try it at chattermate.chat, or read more from our blog.