HyperTeamsWhy it is wrong so plausibly
What you will learn Structurally, why hallucination happens; which situations raise its probability; and how to manage it at organisational scale.
It is not a malfunction
In the previous chapter we said the model's job is producing the natural continuation. A problem follows from that.
"I don't know" is not a natural continuation.
What naturally follows a question is an answer. In human writing, an answer follows a question overwhelmingly more often than "I don't know" does. So the model produces the shape of an answer. The shape gets produced even when there is nothing behind it.
That is hallucination. A property of the structure, not a bug.
What it looks like
1. Inventing things that do not exist
Title, author, publisher, and year all look plausible. Look it up and it is not there. It produced the shape of a book recommendation.
2. Answering things it cannot know
The model does not know your company. But the natural continuation to "how much?" is a number.
3. Mixing something wrong in with things that are right
The most dangerous form.
When two of three lines are right, you believe the third. If everything were wrong you would get suspicious; mixed in, it slips through.
When the probability rises
graph TD
A["Conditions that raise hallucination"] --> B["Areas the model doesn't know<br/>internal info · current facts"]
A --> C["Requests for specific values<br/>numbers · dates · proper nouns"]
A --> D["Questions with a false premise<br/>'when was that clause amended?'"]
A --> E["Requests for long, complex answers"]Watch the fourth (false premise) especially. Ask "when was that clause amended?" about a clause that does not exist, and rather than say it does not exist, it invents an amendment date. When a question carries a premise, accepting the premise is the natural continuation.
How to reduce it (you cannot remove it)
| Method | Effect | Limit |
|---|---|---|
| Add "say you don't know if unsure" | reduces frequency | does not eliminate it |
| Include the source material | large | the answer must be in the material |
| Connect search or database tools | large | requires building |
| Ask for sources too | easier to verify | it can invent the sources |
| Ask twice and compare | finds disagreements | double the cost |
The second and third are the fundamental fixes. Do not make the model fill in what it does not know — give it material and have it answer from within that material.
The second form drops hallucination sharply, because there is nothing to invent.
How to handle it organisationally
An individual can "just check." An organisation needs more than that — you have to design who checks, and when.
Split by risk
| Level | Example | Handling |
|---|---|---|
| Low | polishing, summarising, classification | use as-is |
| Medium | drafting, suggestion lists | use after owner review |
| High | amounts, dates, legal, external sends | human approval required |
These levels have to be built into the workflow. A note saying "please be careful" is not enough. The high-risk path must structurally contain an approval step.
A mechanism for noticing errors
More important still: is there a way to know when it was wrong?
This is what Phase 3 meant by deciding "how you learn it was wrong." If you do not know it was wrong, you cannot measure the effect either.
Common misconceptions
"Won't better models eliminate it?"
The frequency falls. But it is a property of the structure, so it does not reach zero. And as frequency falls people relax, which can make it harder to catch. Rare errors are sometimes more dangerous than frequent ones.
"So we can't use it for anything important?"
You can. But do not have the model generate from memory — give it material and have it answer from within that. And put human approval on the high-risk path. The more important the work, the more structure you use.
Check yourself
1. What does "hallucination is not a malfunction" mean?
Answer
The model's job is producing the natural continuation, and an answer follows a question far more naturally than "I don't know" does. The shape of an answer gets produced even with nothing behind it. It is a property of the structure, so it must be managed rather than removed.
2. Why is two right lines and one wrong line the most dangerous case?
Answer
Because the correct content raises the credibility of the incorrect content. If everything were wrong you would be suspicious; mixed in, it does not get filtered.
3. What fundamentally reduces hallucination?
Answer
Not letting it generate from memory — giving it material and having it answer from within. Change "explain the statute" to "summarise from the text below" and there is nothing to invent. Connecting search and database tools works on the same principle.
That "give it material and attach tools" method is the next chapter → Tool use