Why AI Chatbots Make Things Up — and How to Verify Any AI Answer

AI chatbots make things up because they are built to produce the most plausible next piece of text, not to retrieve verified facts. A language model has no internal database it looks things up in and no mechanism that distinguishes “I know this” from “this sounds like the kind of thing that would be true”. When the plausible answer happens to be correct, you get a fact. When it does not, you get a confident, fluent, completely invented one. The output looks identical either way, and that is the whole problem.

The industry calls this a hallucination, which is a poor name — it implies a malfunction. It is closer to the models working exactly as designed. Understanding that distinction is what turns AI from a thing you either trust blindly or distrust entirely into a tool you can use with judgement.

This article covers why it happens, which kinds of question reliably trigger it, how to spot a fabricated answer without leaving the chat, and a verification routine short enough that you will actually use it.

What is actually happening when an AI invents a fact?

A language model predicts text one piece at a time, choosing each next piece based on patterns learned from an enormous amount of writing. Ask it for the author of a real book and the pattern “this title → this author” was seen often enough during training that the prediction is reliable. Ask it for the author of a book that does not exist, and the model does not hit a wall and stop. It produces the kind of name that would plausibly follow, because producing plausible continuations is the only thing it does.

This also explains the single most unsettling property of hallucinations: the confidence is unrelated to the accuracy. The model’s fluent, assured tone is a learned feature of how explanatory writing sounds, not a signal that it has checked anything. A wrong answer arrives in exactly the same voice as a right one, which is why “it sounded sure” is worthless as evidence.

Which questions reliably produce invented answers?

Hallucinations are not randomly distributed. They cluster around predictable kinds of question, and knowing the clusters is most of the skill. If your question falls into one of these, assume the answer needs checking before you use it.

That last row deserves emphasis because it is the one people cause themselves. Ask “why did the Reserve Bank ban X in 2024?” and a model will often explain why, complete with reasoning, whether or not any such ban happened. The question presupposed the fact, and the model treated the presupposition as given. Phrasing the same query as “did the Reserve Bank ban X in 2024, and if so when?” produces far better behaviour, because it leaves room for the answer to be no.

Fabricated citations are the most consequential category in practice, because they are the hardest to notice. A reference list where every entry is formatted correctly, the journals are real and the authors genuinely publish in that field will pass a glance. Some entries will still be invented. Students and professionals have been caught out by exactly this, and the only defence is opening each link.

Does search-connected AI fix the problem?

It reduces it substantially and does not eliminate it. When a model retrieves live pages and answers from them — the approach behind ChatGPT’s search mode, Perplexity and Google’s AI Overviews — it is grounding its answer in real text rather than in learned patterns, which removes the whole category of invented sources. The citations are real because they were fetched.

Three new failure modes replace the old one, though. The model can misread what the page actually says. It can cite a real page that is itself wrong, which simply launders the error. And it can retrieve a page, then add a sentence of its own that was not in any source — the summary drifts beyond the evidence. So a cited answer is better evidence than an uncited one, but a citation is a pointer to check, not a guarantee.

How do you spot a fabricated answer?

There are tells. None is conclusive alone, but two or three together are a strong signal that you are reading invention rather than recall.

• Suspiciously round or tidy numbers. Real data is messy. “Roughly 70% of businesses” with no source attached is usually a pattern, not a finding.
• Specificity that outruns plausibility. If a model gives you a precise page number, a clause reference and a date for something obscure, ask where it came from. Genuine recall of that detail is less likely than fluent generation of it.
• Sources that are generic. “According to a study by researchers at a leading university” is the shape of a citation with the citation removed.
• Instant agreement when you push back. Tell a model it was wrong when it was right and watch what happens. A model that immediately reverses on any challenge was not anchored to evidence in the first place.
• Internal inconsistency. Ask the same question twice in separate chats. Stable facts come back stable; invented ones drift between answers.

That last test is the most useful single trick in this article. Open two fresh sessions, ask the identical question, and compare. Divergence on a factual point is strong evidence that neither answer is grounded in anything.

What is a 60-second verification routine?

Verification fails in practice because people imagine it means re-researching everything. It does not. It means checking the small number of claims that would actually cost you something if they were wrong.

Step one is where the time saving lives. A 600-word AI draft might contain three claims that matter and 580 words of connective explanation. Checking the three takes a minute. Checking all of it takes an hour, which is why nobody does it and everything goes out unverified.

Step four needs a caveat: different models trained on overlapping data can share the same wrong belief, so agreement does not prove correctness. The asymmetry is what makes the test worth running — agreement tells you little, disagreement tells you a lot.

How do you prompt to reduce invention?

You cannot prompt hallucinations away, but you can make them less likely and much easier to catch. Four habits do most of the work.

1. Give permission not to know. Add “if you are not confident, say so rather than guessing” to the prompt. It is not magic, and it measurably increases how often you get an admission instead of an invention.
2. Supply the source yourself. Paste the document, report or page and ask the model to answer only from it. This is the single most effective change you can make, because it converts a recall task into a reading task.
3. Ask for the reasoning, then check the reasoning. Fabricated conclusions often rest on a step that falls apart when stated explicitly.
4. Avoid leading questions. Ask whether something is true before asking why it is true.

Point two is worth building a habit around. Almost every high-stakes use of AI — summarising a contract, pulling figures out of a report, answering a policy question — can be restructured so the model works from text you provided rather than from memory. Our guide to writing better AI prompts covers the mechanics, and if the document in question is confidential, read whether it is safe to share business data with ChatGPT before pasting it anywhere.

Which tasks are safe, and which are not?

The pattern across that table is simple: AI is reliable when you supply the facts and ask it to work with language, and unreliable when you ask it to supply the facts. Almost every disaster story — the lawyer who filed invented cases, the student with a fabricated bibliography, the article citing a study nobody wrote — is the same mistake in a different costume.

Used with that boundary in mind, these tools are genuinely useful and the hallucination problem becomes a manageable cost rather than a reason to avoid them. If you are choosing between assistants, our comparison of ChatGPT, Gemini and Claude covers which handles which task better. For the publishing side of the question — what happens when AI-written material goes out unchecked — see our guide to AI-generated content and how to spot AI-generated text. More in the technology section.