# Why it is wrong so plausibly > **What you will learn** > Structurally, why hallucination happens; which situations raise its > probability; and how to manage it at organisational scale. ## It is not a malfunction In [the previous chapter](/guide/ai-language-model) we said the model's job is **producing the natural continuation.** A problem follows from that. > **"I don't know" is not a natural continuation.** What naturally follows a question is an answer. In human writing, an answer follows a question overwhelmingly more often than "I don't know" does. So the model **produces the shape of an answer.** The shape gets produced even when there is nothing behind it. That is hallucination. **A property of the structure, not a bug.** ## What it looks like ### 1. Inventing things that do not exist ``` You: Give me 3 books to read on this topic AI: 1. "Data-Driven Organisations" (Kim, 2021, Hanbit Press) ``` Title, author, publisher, and year all look plausible. **Look it up and it is not there.** It produced the shape of a book recommendation. ### 2. Answering things it cannot know ``` You: What was our company's revenue last year? AI: About $4.7 million. ``` The model does not know your company. But the natural continuation to "how much?" is **a number.** ### 3. Mixing something wrong in with things that are right The most dangerous form. ``` You: Explain the statutory annual leave rules AI: Employees with under a year of service accrue 1 paid day per full month worked. (correct) After 3 years, 1 additional day accrues every 2 years. (correct) However, workplaces with fewer than 5 employees are entirely exempt from annual leave rules. (inaccurate) ``` **When two of three lines are right, you believe the third.** If everything were wrong you would get suspicious; mixed in, it slips through. ## When the probability rises ```mermaid graph TD A["Conditions that raise hallucination"] --> B["Areas the model doesn't know
internal info · current facts"] A --> C["Requests for specific values
numbers · dates · proper nouns"] A --> D["Questions with a false premise
'when was that clause amended?'"] A --> E["Requests for long, complex answers"] ``` **Watch the fourth (false premise) especially.** Ask "when was that clause amended?" about a clause that does not exist, and rather than say it does not exist, it **invents an amendment date.** When a question carries a premise, accepting the premise is the natural continuation. ## How to reduce it (you cannot remove it) | Method | Effect | Limit | |---|---|---| | Add "say you don't know if unsure" | reduces frequency | does not eliminate it | | Include the source material | **large** | the answer must be in the material | | Connect search or database tools | **large** | requires building | | Ask for sources too | easier to verify | it can invent the sources | | Ask twice and compare | finds disagreements | double the cost | **The second and third are the fundamental fixes.** Do not make the model fill in what it does not know — **give it material and have it answer from within that material.** ``` ✗ "Explain the statutory annual leave rules" → generated from memory ✓ "Summarise the annual leave provisions in the statute text below. (paste the text)" → extracted from what you gave ``` **The second form drops hallucination sharply**, because there is nothing to invent. ## How to handle it organisationally An individual can "just check." An organisation needs more than that — you have to **design who checks, and when.** ### Split by risk | Level | Example | Handling | |---|---|---| | Low | polishing, summarising, classification | use as-is | | Medium | drafting, suggestion lists | use after owner review | | **High** | amounts, dates, legal, external sends | **human approval required** | These levels have to be **built into the workflow.** A note saying "please be careful" is not enough. The high-risk path must structurally contain an approval step. ### A mechanism for noticing errors More important still: **is there a way to know when it was wrong?** ``` ✗ AI enters it → applied directly → nobody ever knows ✓ AI enters it → flagged on the owner's review list → weekly review ✓ AI enters it → automatically flagged if it differs a lot from the previous value ``` This is what [Phase 3](/guide/ax-phase3-poc) meant by deciding "how you learn it was wrong." **If you do not know it was wrong, you cannot measure the effect either.** ## Common misconceptions ### "Won't better models eliminate it?" The frequency falls. But **it is a property of the structure, so it does not reach zero.** And as frequency falls people relax, which can make it **harder to catch.** Rare errors are sometimes more dangerous than frequent ones. ### "So we can't use it for anything important?" You can. But **do not have the model generate from memory — give it material and have it answer from within that.** And put human approval on the high-risk path. The more important the work, the more structure you use. --- ## Check yourself **1. What does "hallucination is not a malfunction" mean?**
Answer The model's job is **producing the natural continuation**, and an answer follows a question far more naturally than "I don't know" does. The shape of an answer gets produced even with nothing behind it. It is a property of the structure, so it must be managed rather than removed.
**2. Why is two right lines and one wrong line the most dangerous case?**
Answer **Because the correct content raises the credibility of the incorrect content.** If everything were wrong you would be suspicious; mixed in, it does not get filtered.
**3. What fundamentally reduces hallucination?**
Answer **Not letting it generate from memory — giving it material and having it answer from within.** Change "explain the statute" to "summarise from the text below" and there is nothing to invent. Connecting search and database tools works on the same principle.
--- That "give it material and attach tools" method is the next chapter → [Tool use](/guide/ai-tool-use)