A customer tells your bot in message two that the order was a gift. Eleven messages later the bot asks if she'd like to leave a review of her purchase. She screenshots it. It ends up in a Slack channel with a crying-laughing emoji.
Nothing malfunctioned. The bot did exactly what these systems do, and once you see the mechanism, you stop being surprised by the whole category of "it forgot" failures.
What chatbot memory actually is
Here is the uncomfortable truth: a language model has no memory of your conversation. None. Between one reply and the next it retains nothing. Every time a customer sends a message, the application assembles a fresh prompt and sends the whole thing to the model: the system instructions, any retrieved documentation, the previous messages in the chat, and the new one. The model reads all of it from scratch and writes the next reply.
So "memory" is a transcript that gets re-sent, turn after turn. The model rereads it like a stranger handed a case file every thirty seconds.
That has three consequences worth sitting with.
First, the transcript has a size limit, the context window. Once the conversation outgrows it, something has to be cut, and the model never gets a vote on what.
Second, rereading is imperfect. A model can have a fact in front of it and still not use it. More on that below.
Third, nothing carries over between sessions unless you build it. Close the tab, open a new chat tomorrow, and the bot is a blank slate. If it greeted her by name, your application stored the name somewhere and pasted it into the prompt. That's a feature somebody wrote, not something the model remembers.
We've covered the full pipeline in how AI chatbots actually work. This post zooms in on one piece of it, because it's the piece customers notice most.
The three kinds of memory a support bot can have
People say "memory" and mean three different things. Pulling them apart is the most useful thing you can do before you buy or build anything.
Within-conversation memory
This is the transcript. Everything said in the current chat gets re-sent. It works well for short exchanges and degrades in long ones, for reasons we'll get to.
Cross-session memory
The bot "remembers" the customer from last week. This only exists if your system writes something down (a summary, a profile, a list of past tickets) and injects it into the next prompt. It's a database lookup wearing a conversational costume. And it's a privacy decision as much as a technical one: what you store about a person, for how long, and whether they can ask you to delete it.
Knowledge
What the bot knows about your product, your refund policy, your shipping times. This isn't conversational memory at all. It comes from your documentation, fetched at question time through retrieval (RAG for customer support explains how). Mixing this up with chat memory leads teams to the wrong fix. If the bot gave the wrong refund window, that's not a forgetting problem. It's a grounding problem.
Why a bigger window doesn't fix it
Vendors like to announce context windows in the hundreds of thousands of tokens, sometimes a million. The implied promise: nothing will ever be forgotten.
The research says the promise is shakier than the number.
In "Lost in the Middle," Nelson Liu and colleagues at Stanford and elsewhere tested models on multi-document question answering and key-value retrieval. Their finding: models do best when the relevant information sits at the beginning or end of the input, and performance degrades significantly when it sits in the middle, even for models built for long contexts (arXiv 2307.03172).
Chroma's research team took a broader look in a report on what they call context rot. They evaluated 18 models, including GPT-4.1, Claude 4 and Gemini 2.5 variants, and found performance dropping as input length grew. Two details stand out for support use. Even a single distractor passage hurt accuracy against the baseline, and several distractors made it worse. And on LongMemEval, a conversational question-answering benchmark, there was a significant gap between a focused prompt of roughly 300 tokens and the full conversation history of roughly 113,000 tokens, across every model family tested (Chroma, Context Rot).
Read that second result again, because it's the whole argument. Giving the model the entire history performed worse than giving it only the relevant bit. Remembering everything and using what matters are different skills.
Now, be fair about scope. Support chats are rarely 113,000 tokens long. A typical customer conversation is a few hundred words. For most of your traffic, within-chat memory just works, and the long-context research is a warning about where it stops working, not a claim that bots forget everything. We haven't measured where that threshold sits for any particular model, and you shouldn't trust anyone who quotes you a single number for it.
But some chats do run long. The billing dispute with the attached invoice, the angry customer on message forty, the integration question that wanders through three topics. Those are exactly the chats where forgetting costs you the most.
Four ways bots forget, and which one you're looking at
When a customer complains the bot lost the plot, it's usually one of these.
Truncation. The app drops the oldest messages to stay under the limit. The gift detail from message two vanishes because it was the oldest, not because it was unimportant. Early messages often hold the most important facts: who the customer is, what they ordered, what they actually want.
Summarization drift. To save space, some systems compress old messages into a summary. Summaries lose detail by design. A gift becomes "customer asked about an order." Worse, a summary written by a model can introduce a small error that then gets treated as established fact for the rest of the chat.
Attention failure. Everything is still in the prompt, and the model still misses it. This is the lost-in-the-middle pattern. You can't see it in logs, because the information was present. It just wasn't used.
Instruction crowding. The system prompt (tone, rules, escalation policy) sits at the top. In long chats, a heap of conversation lands between those instructions and the newest message, and the bot's behavior can drift from the rules you wrote. We wrote about this in your chatbot's system prompt.
Each has a different fix. Truncation needs smarter selection of what to keep. Summarization drift needs structured notes instead of free-form prose. Attention failure needs shorter, tighter prompts. Instruction crowding needs the rules repeated where they'll be seen. Nobody ships a single switch that handles all four.
Where memory failure becomes a real cost
A forgotten gift detail is embarrassing. A forgotten constraint can be expensive.
The best-known case is Moffatt v. Air Canada. In 2024 the British Columbia Civil Resolution Tribunal held the airline liable after its chatbot told a customer he could apply for a bereavement fare within ninety days of purchase, contradicting the airline's actual policy, which was on a page the chatbot itself linked to. The tribunal said a company is responsible for all information on its website, whether it came from a static page or a chatbot, and awarded roughly $650 CAD plus interest and fees (American Bar Association summary).
That wasn't a memory failure. It was a wrong answer. But the legal logic matters for anything that goes wrong in a long conversation: "the bot forgot" is not a defense. If your bot confirms a refund in message six and denies it in message twenty, your customer will quote message six.
This is why we treat hallucination and forgetting as the same family of problem. In both cases the bot produces fluent text that isn't anchored to the thing it should be anchored to.
What to build around the forgetting
You can't cure it. You can design around it, and the designs are mostly boring.
Keep facts out of the transcript. The most dependable memory is the one that doesn't live in chat history at all. Policy answers should come from your documentation at the moment of the question, with a citation, so a forgotten message can't change them. That's the reason we build ChatterMate to ground answers in your docs rather than in whatever the conversation happened to say.
Capture key details as structured fields. If the customer gives an order number, email, product or deadline, pull it out into a labelled field and re-inject it near the top of each prompt. A short "known so far" block is cheaper and more reliable than hoping the model rediscovers message two. If you've ever read a good support agent's ticket notes, you know the format.
Retrieve narrowly. The Chroma results favor focused prompts over full ones. Fetch the four passages that match, not twenty. We make the same case about retrieval in our post on chunking.
Hand off before the chat degrades. A long, circular conversation is a signal, not a challenge. If the customer has repeated themselves twice, or the bot has asked a question it was already answered, escalate, and pass the transcript and the extracted fields along so the human doesn't start cold. Human handoff that doesn't frustrate customers goes into how.
Test long chats on purpose. Most evaluation uses short, tidy questions. Take your ten worst real conversations, the long ones, and replay them. Check what the bot does at message thirty with a fact from message three. That's a test you can run this afternoon.
Be deliberate about cross-session memory. Decide what you'd store, why, and for how long before you store anything. If a customer would be uncomfortable seeing it on a screen, think twice. Self-hosting helps here because the stored data stays on your infrastructure; we lay out the tradeoffs in self-hosting an AI chatbot.
Is "agent memory" different?
A little. AI agents that take actions (look up an order, issue a refund) often keep a scratchpad of what they've done so far, which is another version of the same trick: write state down, paste it back in. The difference between a chatbot and an agent is about what the system can do, not about whether the underlying model has a real memory. It doesn't. Anyone who tells you their agent "remembers" is describing storage and retrieval, and that's fine, as long as you ask what's stored.
The question to ask a vendor
When someone demos a bot and says it has memory, ask three things. Where is the memory stored? What happens at the 50th message of a chat? And what does the bot do when it isn't sure?
The third matters most. A bot that says "I don't have that detail, let me bring in a person" is infinitely better than one that confidently invents a continuation. Forgetting is survivable. Pretending not to forget isn't.
Written by the ChatterMate team. We build an open-source AI support agent that answers from your documentation, with citations. You can try it free: the first 300 chats cost nothing, and you can self-host it. Start at chattermate.chat. More explainers live in our blog hub.
