Asked who won the IEEE Frank Rosenblatt Award in 2010, a large language model answered "Geoffrey Hinton" and rated itself 93% confident. The correct answer is Michio Sugeno. That example opens a 2025 calibration study from the University of Southern California, and it is the cleanest illustration you'll find of the problem with an AI chatbot confidence score: the number is generated by the same process that produced the wrong answer, using the same broken assumption.
Support teams treat that number as a dial. Set the threshold at 80, escalate anything below it, go to lunch. We've watched people configure exactly that and then wonder why the escalation rate sits near zero while angry tickets keep arriving.
Here is what the number actually is, what it isn't, and what to use instead.
Three different things get called an AI chatbot confidence score
Vendors use one phrase for at least three unrelated measurements. Knowing which one you're looking at changes everything about how much to trust it.
Verbalized confidence. You ask the model, in the prompt, to rate its own certainty from 0 to 100. It writes a number. This is the most common implementation because it's the easiest — no model internals required, works through any API. It's also the weakest. The model isn't consulting an internal register of doubt. It's predicting what a confident-sounding answer looks like, and confident-sounding answers usually carry high numbers.
Token probability. The model's actual output distribution — how much probability mass sat on the tokens it picked. This is a real measurement of something. Unfortunately the thing it measures is fluency, not truth. A fabricated policy stated in smooth English has high token probability. It's fluent. It's also invented.
Retrieval score. In a retrieval-augmented setup, the similarity score between the customer's question and the documents pulled from your help center. This one is genuinely useful, and it's the only one of the three that knows anything about your business. But it tells you whether the retriever found something that looks related. It says nothing about whether the model then used it.
Most dashboards show you one number and don't tell you which. Ask.
The calibration research is not ambiguous
Calibration means what it sounds like: if a system says 80% across a thousand answers, about 800 should be right.
The USC study (Chhikara, 2025) ran six models over SimpleQA, a set of 4,326 short factual questions with unambiguous answers, asking each for an answer plus a confidence score. On open-ended questions GPT-4o got 35.14% right, with an expected calibration error of 0.450. The smaller open models — LLaMA 3 8B, Gemma 2 9B — landed around 5% accuracy with calibration errors near 0.80.
Sit with that second group for a second. Roughly 5% correct, and the gap between stated confidence and reality is about 80 points.
Bigger models were better. Still overconfident. And the paper found the misses cluster in the 80–100% band — exactly the range you'd set a threshold in.
This matches the broader literature. A 2024 survey of confidence elicitation (Xiong et al., ICLR 2024) found verbalized confidence bunching in the 80–100% range regardless of whether the answer was right. Scores collapse toward 0.9 and 1.0, which makes them useless for ranking, which is the one job you wanted them for.
RLHF is part of why
There's a plausible mechanism, and it's slightly uncomfortable. Reinforcement learning from human feedback trains models on which responses people prefer. People prefer confident answers. A hedged, correct answer often loses a head-to-head against a decisive, wrong one.
Work on reward calibration in RLHF (Leng et al., 2024) found that this preference training pushes models toward verbalized overconfidence beyond what their output distributions already do. So the step that makes a model pleasant to talk to also makes it worse at admitting doubt. That's not a bug someone forgot to fix. It's the optimization target doing its job.
The one result worth stealing
Buried in the same USC paper is a finding with immediate practical value, and almost nobody quotes it.
The researchers ran every question twice. Once open-ended. Once with the correct answer hidden among three plausible wrong ones. GPT-4o's accuracy went from 35.14% to 73.42% — no surprise, multiple choice is easier. The calibration error is the interesting part: 0.450 down to 0.037. Near-perfect. The small open models improved too, roughly halving their error.
Constrain what a model is allowed to say and it gets much better at knowing whether it's right.
For a support bot, that points somewhere specific. A free-generating agent staring at your entire help center is in the open-ended condition, which is the badly calibrated one. An agent that first picks which documented answer applies, then writes it up, is closer to the constrained condition. Retrieval already does half of this — the half most teams then throw away by letting the model write whatever it likes on top.
The honest caveat: the study guaranteed the right answer was among the four options. Real customers ask about things you never documented, and no amount of constraint fixes that. But it does suggest the shape of the fix, and the shape is narrower, not smarter.
What actually predicts a wrong answer
Three signals are worth more than any self-reported number. None of them is a magic percentage.
Disagreement across samples. Ask the same question five times at non-zero temperature. If the five answers mean the same thing, the model is on solid ground. If they mean five different things, it's improvising.
This is the intuition behind semantic entropy, published in Nature in 2024 (Farquhar, Kossen, Kuhn and Gal). The important move is that they cluster answers by meaning before measuring spread — "you get 30 days" and "the window is a month" are the same answer wearing different clothes, and a token-level metric would score them as disagreement. Their method detects confabulations, which they define as arbitrary and incorrect generations, without needing task-specific training data.
It costs you several generations per question. For a support bot answering a refund-policy question, that's cheap next to the cost of being wrong.
Retrieval coverage, checked separately from retrieval score. Not "what was the top similarity score" but "did we retrieve a passage that contains an answer at all?" Vector search always returns its nearest neighbours. Ask it about something you've never documented and it returns your five least-irrelevant articles with respectable-looking scores. The retriever cannot return nothing. That's the failure mode: silence would be the honest output, and the architecture can't produce it.
Claim-level attribution. After generation, check each factual sentence against the retrieved passages. Does this specific claim appear in the source, or did the model add it? This catches the dangerous case, which isn't total invention — it's an answer that's 90% grounded with one invented detail in the middle. We've written before about how that mixed answer is worse than an obvious fabrication, because it survives a skim.
What we do, and what we deliberately don't
We build ChatterMate, an open-source AI support agent, so treat this as a stated position rather than a neutral survey.
We don't show customers a confidence percentage. We considered it. The reason we didn't is that a number on a screen implies a calibration we can't honestly claim — and the moment it's visible, someone builds a workflow on it.
What we gate on instead is attribution. Every answer is grounded in retrieved documentation and carries citations back to the source article. If the retrieval step doesn't return something the answer can stand on, the agent's correct move is to say so and hand the conversation to a human, rather than produce fluent text with a "72%" next to it.
The honest limitation: a citation proves a passage was retrieved. It does not prove the model read it correctly. Attribution is a stronger signal than self-reported confidence, not a guarantee. Anyone telling you they've solved this has skipped a step.
Setting an escalation threshold without guessing
If your platform already exposes an AI chatbot confidence score, you're not stuck with it. You just can't trust the vendor's default threshold, because it was never measured against your content.
Pull 200 real conversations and label them. Right, wrong, or partially wrong. This takes an afternoon and it is the single highest-value thing a support lead can do with AI this quarter. Everything below depends on it.
Plot the score against correctness. Bucket by stated confidence — 50s, 60s, 70s, 80s, 90s — and compute actual accuracy per bucket. You now have your own reliability diagram. Most teams find the curve is nearly flat, which means the score carries almost no information and no threshold will save them. That's a finding, not a failure. Better to know.
Price the two errors. Escalating an answer the bot would have got right costs you agent minutes. Shipping a wrong answer costs you a refund dispute, a bad review, sometimes a regulator. Those are not equal, and your threshold should reflect the ratio rather than a round number someone liked.
Track abstention as a metric you want. If your bot never says "I don't know", that is not a sign it's good. It's a sign it has no mechanism for doubt. We'd argue an honest abstention rate belongs on the dashboard next to containment rate — and that a team reporting 95% containment with 0% abstention is reporting one number twice.
Re-run it after every change. New model version, new prompt, new batch of help articles: the calibration you measured is no longer the calibration you have. This is the same discipline as an eval set, and it's the same reason people skip it.
The uncomfortable version
A confidence score is a system grading its own homework in a subject it doesn't know it's failing. The research says it grades generously and clusters its marks at the top.
You can still build a support bot you'd stake your brand on. It just doesn't come from finding the right threshold on the wrong number. It comes from grounding answers in documentation you control, checking that the grounding held, and treating "escalate to a human" as a correct outcome rather than a defeat.
So: go and find out what your bot's number is worth. Two hundred conversations, one afternoon. You'll either gain a tool you can trust or lose one you shouldn't have been trusting.
Written by the ChatterMate team — we build an open-source AI support agent that grounds every answer in your own documentation and cites its sources. It's free to start (first 300 chats free) and self-hostable if you'd rather the whole thing ran on your own hardware. More in our blog.
