A customer writes to your support bot in Portuguese. Your help center is in English. The multilingual AI chatbot you bought answers in fluent, confident Portuguese, and the answer is wrong, because it never found the right English article in the first place.
That failure is common, and it's invisible from the outside. The reply sounds perfect. Nobody flags it.
So when teams say they want a multilingual AI chatbot, they usually picture the language model's side of the job: can it write in Portuguese? Yes, almost any modern model can. The harder question sits one step earlier. Can the system find the right passage when the question and the document are in different languages?
We build ChatterMate, an open-source AI support agent, so we spend a lot of time looking at where bots go wrong. This is the part of multilingual support we think gets the least attention. Here's how it works, what the research says, and how to check your own bot in an afternoon.
Why a multilingual AI chatbot is a retrieval problem
Most support bots built on your docs work in two stages. First, a retriever searches your knowledge base and pulls the few passages most likely to contain the answer. Then a language model reads those passages and writes the reply. We've covered the mechanics in our post on RAG for customer support.
The language model is the part people test. Type a question in Spanish, get Spanish back, tick the box. But the model can only work with what the retriever hands it. If the retriever returns the wrong passages, a fluent model will write a fluent answer from them. Or it will fall back on whatever it half-remembers from training, which is how you get a hallucinated refund policy in perfect German.
So the question to ask a vendor, or your own team, is not "does it support 40 languages?" It's "does retrieval work across languages, and how do you know?"
What the research actually shows
Researchers have started to measure this. A paper called "Investigating Language Preference of Multilingual RAG Systems" (Findings of ACL 2025) looked at how retrievers rank documents when the query and the document are in different languages. The paper is on arXiv.
Three findings matter for a support team.
Retrievers favour documents in the query's own language. The authors built a metric, MultiLingualRankShift, that measures how much a document's rank improves once it's translated into the query language. Matching-language passages scored highest, and cross-language matches usually scored lower. English was the notable exception: non-English queries tended to do better against English documents than against documents in other foreign languages. In their bge-m3 runs, the English monolingual score was 56.03, against 41.15 for Korean and 39.19 for Portuguese.
The document's language matters more than the query's. Passages written in high-resource languages ranked higher than those in lower-resource ones. If part of your help center is in a smaller language, that content is at a disadvantage before anyone asks a question.
Better retrieval preference doesn't reliably mean better answers. For English queries, answer quality tracked retriever preference fairly well. For non-English queries the link was weak. In the authors' tests, the best results often came when passages matched the query language, not when everything was funnelled into English.
One caution before you quote any of this. The experiments used Wikipedia-based question sets (2,827 queries) and a limited set of languages. Support content is messier than Wikipedia: product names, error codes, informal phrasing. The paper says plainly that generalisation is untested. Treat it as evidence of a mechanism, not a benchmark for your help center.
Embedding models are better than they were, and still not magic
The usual fix is a multilingual embedding model, one that maps text in many languages into the same vector space so that "reembolso" and "refund" land near each other. These have improved a lot. Microsoft's multilingual E5 report, for instance, describes models trained on roughly a billion multilingual text pairs and evaluated on MIRACL, a retrieval benchmark across 16 languages. Its large model averaged 66.5 nDCG@10 on the MIRACL dev set, well above BM25 at 39.3. The report is on arXiv.
That's a real gain. But read what it measures: retrieval within each language, on Wikipedia-style text. It says less about a Portuguese customer typing "meu pedido não chegou" against an English article titled "Delayed shipment troubleshooting." The vector space is shared, but it isn't perfectly aligned, and the gap is widest for the languages with the least training data. Our post on chatbot embeddings goes into why two sentences with the same meaning can still sit far apart.
So we'd put it this way. A good multilingual embedding model makes cross-language retrieval workable. It doesn't make it safe to skip testing.
Three ways bots break in other languages
These are the failures we'd look for first. We haven't measured how often each happens in production, so read them as a checklist, not statistics.
1. Right language, wrong source
The bot replies in the customer's language but cites the wrong article, or no article. This is the failure from the opening. The only way to catch it is to look at which passages were retrieved, not just at the reply. If your bot shows sources, check that the cited page actually answers the question.
2. Silent fallback to the model's memory
When retrieval returns nothing useful, some bots answer anyway. In English that might get caught by a confidence threshold. In another language, the scores may be systematically lower, so either the bot refuses far more often than it should, or the threshold gets loosened and it starts guessing. Either is bad. We wrote about thresholds in chatbot confidence scores.
3. Terms that shouldn't be translated
Your product has names. Plan names, feature names, button labels, error codes. If a model helpfully translates "Workspace Settings" into Spanish and your interface says "Workspace Settings," the customer can't find it. Same in reverse: if your Spanish docs call it "Configuración del espacio," an English-speaking agent reading the transcript has no idea what the customer meant. Decide which strings stay fixed, and say so in the prompt.
A one-afternoon test for your own bot
You don't need a benchmark. You need about fifty real questions.
- Pull real questions. Take 25 to 50 from your actual inbox, covering the topics customers ask about most. Shipping, billing, cancellations, login problems.
- Translate them into your top three or four customer languages. A native speaker is best. If you don't have one, use a translation tool and then have someone fluent skim for anything that reads oddly, because real customers don't write like translation output.
- Run each one through the bot and record the sources it retrieved, not just the reply. Mark each as: right source, wrong source, no source.
- Compare against English. If English gets 90% right and Portuguese gets 60%, you've found your gap, and you know which language to fix first.
- Add messy versions. Typos, no accents, mixed languages ("mi order no llegó"). People write like this, and bots that pass the clean test often don't pass this one. Our guide to testing an AI chatbot covers how to keep a standing set of these.
Keep the spreadsheet. You'll want to rerun it every time you change the knowledge base, the model, or the retrieval settings.
What to do when the gap is real
There are a few levers, in rough order of cost.
Translate the content that matters most. You don't need your whole help center in eight languages. The top 20 articles by traffic cover most questions. Translate those first, and keep them in sync with the English originals. Stale translations are worse than none, since they confidently state last quarter's policy.
Change how you chunk and index. Passage size and structure affect retrieval in every language, and short, self-contained passages tend to hold up better across languages. See chunking strategy for support bots.
Translate the query, not the corpus. Some pipelines translate the customer's question into the document language before retrieval. The research above suggests this paid off for some language pairs, with a cost: an extra step, extra latency, and a chance the translation itself drifts. Test it against your fifty questions before adopting it.
Route low-confidence cases to a person. This is the backstop. If a bot can't find a source, it should say so and hand off, in the customer's language, with the conversation attached. Our post on human handoff covers what a good handoff looks like. If nobody on your team reads Japanese, that handoff needs a plan too: a translated summary, an honest wait time, or a message saying which hours a Japanese speaker is online.
Does it matter enough to bother?
Fair question. Many small teams serve one language and can ignore all of this.
But if you sell across borders, language is rarely a nice-to-have. CSA Research, a firm that studies the localisation industry, surveyed 8,709 consumers in 29 countries in 2020. Its summary reports that around two thirds said they'd choose a similar product if the information came in their own language. CSA works in translation, so it has an interest in the answer, and we're citing its published summary, not the full report. Still, the direction matches what most support teams see in their inbox: people ask for help in the language they think in.
And there's a quieter cost. A bot that handles English well and other languages badly doesn't look broken. It looks like a bot that's fine for most people. The customers it fails are the ones least likely to tell you.
Where this leaves a small team
Pick the two or three languages that make up most of your non-English traffic. Build the fifty-question test. Fix retrieval before you touch the prompt, and make the handoff work in each language. That's a week of effort, not a quarter.
If you run your own stack, self-hosting gives you a practical advantage here: you can swap the embedding model and rerun the same test on your own data, without waiting for a vendor to change it. We haven't benchmarked ChatterMate's multilingual retrieval against other tools, so we won't tell you it's better. We'd tell you to run the test on whatever you choose, including us.
ChatterMate is open source, free to start with your first 300 chats, and self-hostable. You can try it at chattermate.chat and point it at your own docs and your own fifty questions.
Written by the ChatterMate team. We build an open-source AI support agent.
