In 2022 a man asked Air Canada's website chatbot about bereavement fares. The bot gave him a policy that didn't exist. He booked on it, got refused a refund, and took the airline to a tribunal in British Columbia. Air Canada argued, with a straight face, that the chatbot was a separate legal entity responsible for its own actions. The tribunal disagreed and ordered a payout of roughly CAD 650 plus interest and fees (American Bar Association summary). That is what an AI chatbot rollout with no brakes can cost.
The money was small. The principle wasn't: whatever your bot says is something you said. So the safer way to launch is in stages, not on a single launch day.
Written by the ChatterMate team. We build an open-source AI support agent, so we have a stake here. But the plan below works with any tool.
Why the big-bang launch is the wrong default for an AI chatbot rollout
Most teams do it the same way. Connect the help docs, click around the widget for twenty minutes, paste the embed code, announce it. Then they find out what the bot does with real customers, in front of real customers.
Real customers don't ask the questions in your FAQ. They ask three questions in one message. They paste an order number with a typo. They ask whether you'll refund them because the courier lost the parcel, a question your docs answer only by implication. The bot has to improvise, and improvising is exactly where wrong answers come from.
So don't give it that test live. Give it four easier ones first. Each stage exposes the bot to a little more risk and has one exit criterion you write down before you start, so you can't talk yourself into moving on because the deadline is Friday.
Stage 1: Replay (the bot talks to nobody)
Pull 150 to 200 real tickets or chats from the last few months. Run the customer's first message through the bot, offline, and read what comes back next to what your team actually said.
That number isn't a benchmark. It's the point where you stop being surprised. With fewer than about 100 you'll miss whole categories; past 200 you're mostly reading the same refund question again.
Sort each answer into one of four buckets: correct, correct but badly worded, wrong, and confidently wrong. The last bucket is the only one that should worry you. A bot that says "I don't know, let me get a person" is working. A bot that invents a returns window is a lawsuit with a typing indicator.
Check that every factual claim points to a source. In ChatterMate answers carry citations back to the doc they came from, which makes this stage fast: you click, you read the paragraph, you know in ten seconds whether the bot read it right or made a leap. If your tool can't show its sources, replay takes much longer, and that tells you something about the tool.
Exit criterion: zero confidently-wrong answers on anything involving money, dates, legal terms or account access. Not "under 2%." Zero. Fix the docs (it's nearly always the docs, see our piece on training an AI chatbot on your docs) and replay again until the count is zero.
Stage 2: Draft mode (the bot talks, a human sends)
Now put the bot in the live inbox, but invisible. It drafts a reply for each incoming conversation. A person reads the draft, edits it or throws it away, and sends it.
This is the stage people skip because it feels like work without payoff. It's the opposite. You collect something nobody else has: a running record of how often your team accepts the draft untouched, how often they rewrite it, and how often they bin it. Those three numbers are your real accuracy, measured on real traffic, at the price of a few minutes a day.
Staged approaches like this have a vendor-neutral shape. One AI-agent consultancy describes the same progression as three steps: draft-and-approve, then limited autonomy inside a narrow set of times, action types or customer segments, then full autonomy per action type. The progression is "action by action," not system-wide, and it's reversible (BotsCrew on silent mode). They cite one client that used roughly 90% field-level accuracy as the graduation bar, alongside stakeholder sign-off. Take the 90 as one example, not a law. The point is that someone picked a number in advance.
Run draft mode for at least two weeks, long enough to see a Monday rush and a quiet Thursday.
Exit criterion: agents accept the draft with little or no editing on the large majority of the topics you plan to open up first. You decide "large majority." Write it down now.
Stage 3: Narrow live (a slice of customers, a slice of topics)
Finally, customers talk to the bot directly. But not all of them, and not about everything.
There are two dials. One is who: ten percent of traffic, or only visitors on the pricing page, or only people outside business hours. The other is what: let it answer order status and shipping and nothing else, and route everything outside that list straight to a human. Twig, a vendor in this space, suggests testing threshold changes on about 10% of traffic and setting stricter confidence bars for riskier topics, in the 80-85% range for low-risk questions up to 92-95% for high-risk ones (Twig's testing guide). Treat those as starting points to argue with, not published standards. The idea underneath them is sound, though. A wrong answer about opening hours and a wrong answer about a refund are not the same size of mistake, so they shouldn't clear the same bar. We go deeper on this in what a confidence score actually tells you.
The part that decides whether this stage works is the handoff. When the bot is unsure, the customer needs a person within seconds and shouldn't have to repeat the question. If handoff is clumsy, a narrow launch just produces a smaller group of annoyed customers. Our human handoff playbook covers the mechanics.
Keep two things visible on a dashboard from day one: the share of conversations the bot closed without a human, and the share where the customer asked for a person anyway. The first number going up while the second goes up with it is not success. It's the bot blocking people. We wrote about why in our post on AI containment rate.
Exit criterion: four straight weeks (Twig suggests two to four of monitoring before you adjust anything) with no confidently-wrong answers surfaced by customer complaints or by your own sampling, and no rise in "talk to a human" requests.
Stage 4: Widen, one dial at a time
Move one dial per week. Add a topic, or raise the traffic share. Never both. If something breaks, you know which change did it.
This is boring on purpose. It's also where most of the value shows up, because by now the bot has a track record and you have the evidence to defend turning it up. When someone on the leadership team asks why you didn't launch it everywhere in week one, you have a spreadsheet instead of a feeling.
Keep a standing sample going even after full rollout. Read twenty conversations a week. Models change underneath you, docs go stale, products get new pricing. A bot that was right in October can be quietly wrong in March. Our piece on AI chatbot testing shows how to turn that weekly read into a regression set so the same mistake can't come back twice.
The kill switch, written down before launch
Every stage needs an answer to one question: how do we turn this off in under a minute, and who's allowed to?
Write down the name. Write down the button. Test it once on a Tuesday afternoon when nothing is wrong, so it isn't the first time you've pressed it when something is. If the answer is "we'd have to ask the developer," you aren't ready for stage 3.
A fallback matters just as much: when the bot is off, what does the widget show? A contact form and a promised response time beats a dead widget.
A one-page version you can copy
| Stage | Who sees the bot | Exit criterion |
|---|---|---|
| 1. Replay | Nobody | Zero confidently-wrong answers on money, dates, legal, account topics |
| 2. Draft mode | Your agents only | Drafts accepted with little editing on the topics you'll open first |
| 3. Narrow live | A slice of traffic, a short topic list | Four clean weeks, no rise in handoff requests |
| 4. Widen | More traffic or one new topic per week | Weekly sample of 20 conversations stays clean |
Total time from replay to full rollout for a small team: somewhere between six and ten weeks. That sounds slow until you compare it with the cost of finding out in public.
What we'd do differently if we were starting over
We'd skip the instinct to launch with a long list of topics because the demo looks more impressive. A bot that handles three things perfectly beats one that handles thirty things at 85%. Customers forgive a bot that says "that's outside what I can help with, here's a person." They don't forgive one that sounds certain and is wrong.
And we'd stop treating the first weeks as a launch. They're a trial, with an exit and a record of the results.
Try it on your own docs
ChatterMate is open source, free to start (the Free plan includes 50 AI replies a month), and you can self-host it. Point it at your help docs, read what it says in your own replay, and see where it holds up before a single customer sees it. Start at chattermate.chat.
More on running AI support well is on our blog hub.



