In June 2023, a New York federal court sanctioned two lawyers and their firm $5,000. Their brief cited six court decisions that did not exist. ChatGPT had produced them — complete with plausible case names, fabricated quotations, and internal citations to other cases that were also invented. When the court asked for copies, one of the lawyers filed excerpts. Those were fabricated too.
The useful detail is not that the model lied. It is that when asked to produce the documents, it produced convincing ones. The model was not malfunctioning at that point. It was doing exactly the thing it does.
The one-sentence version
A language model does not look facts up. It predicts what text should come next, and a fact it has seen and a fact it has never seen produce the same kind of output — confident, fluent, correctly formatted prose.
That is the whole problem. There is no internal boundary between “recalling” and “inventing”, because from the model’s perspective those are not different operations.
Why “hallucination” is a slightly annoying word
The term has stuck, and it is what everyone searches for, but it implies the model is perceiving something that is not there. It is not perceiving anything. Some researchers prefer confabulation — the term for a person filling a memory gap with plausible invention, sincerely and without any awareness of doing it. That is much closer to what happens.
Cambridge Dictionary added the AI-specific sense of “hallucinate” in 2023, so the word has won. But if you keep confabulation in mind, you will predict the behaviour better.
What actually causes it
Four things, roughly in order of how often they matter to you.
1. The gap between “knows” and “sounds right”
Models are trained to produce likely text. “Likely” and “true” overlap heavily — which is why this works at all — but they are not the same target. When a question falls into a gap, the most likely continuation is still a fluent, confident answer, because that is what the training data looks like. Reference texts are rarely written in a hedging tone.
This is why hallucinations are so often specific. A vague answer would be less likely than a precise one, so you get a precise one: a page number, a date, a case citation. The specificity is generated by the same process as the rest.
2. Answering is rewarded more than declining
If a model is scored on whether it produced a satisfying answer, guessing beats abstaining. A guess is sometimes right. “I don’t know” is never scored as right. Unless the training explicitly rewards appropriate uncertainty, the incentive points toward always producing something.
You can watch this happen. Ask about a real but obscure thing and you will usually get a confident answer. Ask about a thing you just invented and you will often get a confident answer about that too.
3. It cannot check itself
The model has no separate store of facts to compare its output against. Anthropic published research in 2025 describing internal circuitry in Claude that appears to govern whether the model treats a name as familiar — and cases where that mechanism misfires, suppressing the “I don’t know this” response and letting a plausible answer through instead.
That is a mechanistic account of something you can observe from outside: the failure is not the model knowing it is wrong and saying it anyway. There is no moment where it knows.
4. Everything outside the context window is reconstruction
A model has two sources: the weights (compressed, lossy, frozen at training) and whatever is in the current conversation. Anything not in front of it is reconstructed from a lossy summary of a large amount of text.
Details that were rare in the training data — a specific page number, an exact quotation, the pricing of a small product — are exactly the details most likely to be reconstructed wrongly. This is also why what fits in the context window matters so much: material inside it is being read, material outside it is being remembered.
AI agents working through long tasks run into the same limit from the other side. As the window fills, their own earlier steps get dropped or summarised, so they end up remembering their work rather than reading it.
What it looks like in the wild
Fabricated citations are the classic, but the pattern is broader.
| Type | Example | Why it is dangerous |
|---|---|---|
| Invented sources | A paper, case, or book that does not exist | Formatted perfectly, so it passes a glance |
| Real source, wrong claim | An actual study cited for a finding it does not contain | Survives a check that the source exists |
| Fabricated quotations | Words attributed to a real, named person | Defamation risk, and very hard to spot |
| Confident wrong numbers | Prices, limits, statistics, dates | Read as fact because they are precise |
| Invented product features | A capability the tool does not have | Fails only when someone tries it |
| Plausible fake APIs | A function that should exist but doesn’t | Wastes an afternoon |
Some real consequences, for scale. In February 2024, a tribunal ordered Air Canada to pay damages after its support chatbot described a bereavement fare policy the airline did not offer — the airline argued unsuccessfully that it should not be responsible for its own chatbot. In October 2025, Deloitte’s report for the Australian government, worth A$440,000, was found to contain non-existent academic sources and fabricated court quotes. A month later, a CA$1.6 million health workforce plan from the same firm was found to cite four research papers that did not exist.
The organisations involved were not careless amateurs. The output is convincing precisely because being convincing is what it is optimised for.
What actually reduces it
In rough order of effect per unit of effort.
Give it the source material. The single biggest improvement. A model summarising a document you pasted in is doing a fundamentally more reliable task than a model recalling one. This is the entire premise of retrieval-based systems, and you get most of the benefit manually by pasting the thing in.
Ask for the source, then check it exists. Not “cite your sources” — models will happily generate citations. Ask for the URL, then open it. The failure mode you are hunting is a real-looking reference to nothing.
Prefer questions with verifiable answers. “What does this contract say about termination” is checkable against the contract in front of you. “What is standard in contracts like this” is not checkable against anything, and is where invention lives.
Give it a way out. Explicitly permitting “I don’t know” measurably changes behaviour. Something like “If you are not confident, say so rather than guessing” is worth the eleven words.
Ask twice, separately. Not “are you sure?” — that mostly produces an apology and a new fabrication. Ask the same question in a fresh conversation and compare. Stable facts tend to be reproduced; invented ones tend to drift.
Watch for the specificity spike. When an answer suddenly becomes very precise — a page number, an exact figure, a named clause — that is where to look first. Precision is not evidence of retrieval.
Never skip the check on anything that matters. If a wrong answer is expensive, a human verifies it. Every case in this article was a system that worked fine right up until nobody checked.
A short fact-checking routine covers the mechanics.
What does not work
- “Don’t hallucinate.” The model cannot comply, because it cannot detect the state you are asking it to avoid.
- “Are you sure?” Produces agreement or an apology depending on tone, not verification.
- Assuming search or browsing fixes it. It helps a lot — the model is reading rather than recalling. But it can still misread a page, cite a source that does not support the claim, or blend a retrieved fact with a remembered one.
- Assuming newer or larger models fixed it. They hallucinate less, and this has improved meaningfully. The failure mode is intrinsic to how the systems work, and a rarer failure that you have stopped checking for is arguably more dangerous than a common one you expect.
The thing worth remembering
Treat model output as a confident draft from someone who has read an enormous amount, remembers most of it approximately, will never tell you which parts are approximate, and does not experience the difference between recalling and inventing.
You would find that person extremely useful. You would also check their citations.
Sources: Mata v. Avianca court docket (Justia) · Seyfarth Shaw on the sanctions · Hallucination (artificial intelligence) — Wikipedia, which collects the Air Canada and Deloitte incidents. All accessed 6 September 2026.