A customer types "why was I charged twice in July" and hits enter. Two seconds later there's an answer on screen. Most explanations of how AI chatbots work stop at "it uses AI" — which tells you nothing about why the same bot nails that question on Monday and invents a refund policy on Tuesday.
Six things happen in those two seconds. Each one can fail on its own, and each one fails in a different way. If you run support and you're evaluating bots, or you already have one and it's behaving strangely, the pipeline below is the thing you actually need to understand. Not the model. The pipeline.
How AI chatbots work, stage by stage
Here's the whole chain, then we'll take it apart:
Message → embedding → retrieval → prompt assembly → generation → tools & handoff.
A modern support bot is not one system. It's five or six systems in a trench coat, and the language model is only the fifth. Most of the failures people blame on "the AI" happen in stages two and three, before the model has seen anything.
1. The message arrives (and it's messier than you think)
The raw input is rarely a clean question. It's "hey so the thing didn't work again?? order 44821" — typos, missing context, a reference to a conversation from three weeks ago, sometimes a screenshot.
Before anything else, the system decides what the message is about. Some platforms run an intent classifier here. Others skip classification entirely and pass the text straight to retrieval. There's also the question of history: does the bot include the previous eight turns of the conversation, or just this message? Include too little and the bot loses the thread on follow-ups like "and the other one?" Include too much and you drag irrelevant text into every downstream stage.
Intent classifiers used to be the whole architecture. You'd define forty intents, train a classifier, and route each one to a scripted reply. That approach is mostly gone now, and for a good reason: customers ask things you never enumerated, and an intent list has a hard edge. Retrieval doesn't. We covered the trade-off properly in rule-based chatbots vs AI chatbots, but the short version is that classification gives you predictability and retrieval gives you coverage, and support volume rewards coverage.
This stage is unglamorous plumbing and it decides more outcomes than the model choice does.
2. Embedding: turning words into coordinates
The bot converts the question into a vector — a list of a few hundred or few thousand numbers that positions the meaning of that sentence in a mathematical space. Your help docs went through the same process ahead of time, chopped into chunks and embedded one by one.
Why bother? Because keyword search fails on the way real customers write. Someone asking "can I get my money back" will never match a doc titled "Refund eligibility window" on shared keywords. In vector space those two sit close together, because the embedding model learned that they mean roughly the same thing.
The catch is that "roughly the same thing" is doing a lot of work. Embeddings are lossy. A chunk that says "customers on the legacy plan cannot self-serve refunds" and one that says "customers can self-serve refunds" land almost on top of each other — the negation barely moves the vector. Retrieval is happy. The answer is wrong.
3. Retrieval: finding the right paragraph
Now the system searches the vector index for the closest chunks to the question and pulls back the top handful. This is the retrieval step in RAG, and it decides more about answer quality than any other stage in the chain.
Anthropic published hard numbers on this in September 2024. On their evaluation, a standard embeddings-plus-BM25 setup failed to surface the right chunk in the top 20 about 5.7% of the time. Adding chunk-level context before embedding cut that failure rate by 35%, down to 3.7%. Combining contextual embeddings with contextual BM25 cut it by 49%, to 2.9%. Layering a reranker on top took the reduction to 67% (Anthropic, Contextual Retrieval in AI Systems).
Read that again, because the framing matters. Even the good configuration misses roughly one time in fifty. And when retrieval misses, the model downstream has no idea it missed. It just answers from whatever it got.
Chunking is where most teams lose accuracy without noticing. Split your docs at 500 characters and a policy sentence gets severed from the condition that qualifies it. Split at 4,000 and every retrieved chunk arrives padded with three paragraphs of unrelated text. We've seen the same knowledge base go from useless to reliable with no model change at all — just better chunk boundaries and a heading prepended to every chunk so it carries its own context. If you're setting this up from scratch, our guide on training an AI chatbot on your docs covers the chunking decisions in more detail.
4. Prompt assembly: the part nobody talks about
The retrieved chunks get stitched into a prompt alongside the system instructions, the persona, the conversation history, and whatever tool definitions are available. This bundle is what the model actually sees. The customer's question might be 2% of it.
Order matters here in a way that surprises people. Nelson Liu and colleagues at Stanford tested this directly: they buried the answer-bearing document at different positions in a long context and measured accuracy. Performance was highest when the relevant information sat at the beginning or the end, and dropped noticeably when it sat in the middle — even for models explicitly built for long contexts (Lost in the Middle: How Language Models Use Long Contexts, TACL 2024).
So a bot that stuffs twenty chunks into the prompt can perform worse than one that carefully picks four. More context is not more accuracy. This is why reranking exists, and why "we have a million-token context window" is a weak answer to "how do you make sure it finds the right doc".
5. Generation: the model writes
Only now does the language model do its bit. It reads the assembled prompt and produces text token by token, each one conditioned on everything before it.
Two things are worth being precise about. First, the model is not looking anything up while it writes — the lookup already happened in stage three. Second, the model has no internal flag for "I wasn't given the answer." A well-instructed model will say it doesn't know. An under-instructed one will produce a fluent, confident, entirely invented refund policy, because generating plausible text is exactly what it was trained to do.
There's also a reason the answer appears word by word rather than all at once. Generation is sequential, so the first token is available long before the last one, and streaming it to the widget hides most of the wait. It's a UX trick, not a speed improvement — the total time is the same. But a customer watching text appear will wait several seconds without complaint, where a blank box for the same duration reads as broken.
This is the whole argument for grounding. The original RAG paper from Patrick Lewis and co-authors at FAIR and UCL made the point back in 2020: pairing a model's parametric memory with an explicit, non-parametric memory produces "more specific, diverse and factual language" than the model alone, and — the part people skip — it gives you provenance (Lewis et al., NeurIPS 2020). You can point at the paragraph the answer came from.
That's the design decision we made in ChatterMate. Answers are grounded in your documents and every answer carries its citation, so a support lead can audit any conversation and see which doc produced it. It doesn't make hallucination physically impossible — nothing does — but it converts a silent failure into a visible one. If you want the longer version of that argument, we wrote about why AI chatbots hallucinate and about RAG for customer support specifically.
6. Tools and handoff: where a chatbot becomes an agent
Everything above answers questions. It doesn't do anything.
"Why was I charged twice in July" cannot be answered from documentation. It needs a call to your billing system with that customer's ID. A bot with tool access can make that call, read the result, and put the real charge history in front of the customer. A bot without it can only explain your billing policy in general terms — which is the polite version of no answer at all.
This is the actual line between a chatbot and an agent, and it's blurrier in marketing than in code. We've written a full breakdown of AI chatbot vs AI agent if you want to pull that thread.
Then there's the exit. Every bot needs a route to a human, and the route is a design problem, not a fallback. Handing off without passing the transcript, or handing off to an empty queue at 11pm, is worse than never offering the bot. Our post on chatbot human handoff goes into what a good one looks like.
What this means when you're evaluating a bot
Once you see the pipeline, vendor demos get easier to read. The demo always shows stage five — the fluent answer. The stages that determine whether you get that answer on your own messy content are three and four, and nobody demos those.
Questions that get you past the demo:
- How do you chunk my documents, and can I see the chunks?
- How many chunks go into the prompt, and do you rerank them?
- What does the bot do when retrieval returns nothing relevant?
- Can I see which document produced any given answer?
- What triggers a handoff, and what does the human receive?
That fourth one is the tell. If a vendor can't show you the source paragraph behind an answer, they haven't built provenance in, and you have no way to debug a wrong answer other than reporting it and hoping.
The uncomfortable part
Here's the thing most of these explainers won't say. A support bot's ceiling is your documentation. All six stages are machinery for finding and rephrasing text that already exists somewhere in your content. If the answer to "why was I charged twice in July" lives only in the head of one person on your billing team, no amount of model upgrading will surface it.
Which means the highest-return work in deploying a support bot usually isn't the deployment. It's the two weeks beforehand: pulling your last 500 tickets, finding the twenty questions that come up constantly, and writing real answers to the eight that were never documented.
Boring. Effective. Skipped by almost everyone.
The pipeline is the part you can inspect, and now you know what to inspect. Start at retrieval, not at the model — that's where the answers actually get found or lost.
Written by the ChatterMate team — we build an open-source AI support agent that answers from your docs and cites its sources. You can try it free at chattermate.chat (first 300 chats are on us), or self-host the whole thing if you'd rather your customer conversations never leave your infrastructure. More explainers in our blog.
