# Things to watch out for > **What you will learn** > Three limits of AI, and how to decide what you must verify. This chapter is not here to scare you. **Knowing how far you can trust it actually makes it easier to use.** It is like knowing where your car's blind spots are — that is what lets you drive relaxed. --- ## Limit 1 — It invents things, plausibly The most important limit. ### What happens When AI meets something it does not know, it sometimes **does not say "I don't know" and instead produces something plausible.** This is usually called **hallucination**. The problem is that **a wrong answer arrives with exactly as much confidence as a right one.** You cannot tell them apart by tone. ### How it actually shows up ``` You: What was our company's revenue in 2023? AI: Revenue in 2023 was approximately $4.7 million. ``` AI **cannot possibly know your company's revenue.** And yet a number appeared. In the course of producing something shaped like an answer, it filled in a plausible figure. Or this: ``` You: Give me 3 books worth reading on this topic AI: 1. "Data-Driven Organisations" (Kim, 2021) 2. ... ``` **Those books may not exist.** Title, author, and year all look plausible; look them up and they are not there. ### Why this happens In [what AI is](/guide/ai-what-is-it) we described AI as "a program that got very good at guessing the next word." **A plausible answer is a more natural continuation than "I don't know,"** so sometimes it goes that way. ### What to do about it | For this | Do this | |---|---| | Numbers, dates, statistics | **always verify** | | Names, book and paper titles, legal citations | **always verify** | | Internal company information | it cannot know. You have to supply it | | Polishing, summarising, translating | generally safe (it works on what you gave it) | | Suggesting ideas | safe (you are choosing anyway) | **Rule of thumb**: work that processes material you provided is safe; **work where AI has to fill in information it does not have is risky.** > **One thing that helps** > Adding "if you're not sure, say you don't know" reduces how often it invents > things. It does not eliminate it. --- ## Limit 2 — It may not know current information ### What happens AI is built from material up to a certain point in time. It does not know what happened afterwards. ``` You: What's today's exchange rate? AI: (doesn't know, or gives an old value, or invents one) ``` ### What to do about it - **Facts about today** (rates, weather, news, share prices) → use search - **Rules or regulations that recently changed** → verify - **Things that rarely change** (Excel functions, writing, general knowledge) → fine That said, **AI with tools attached is different.** Connect it so it can search or look through internal material directly and it can answer with current information. That is [tools](/guide/mcp-why-standard), covered later. At this stage, assume "by default it does not know." --- ## Limit 3 — Before you paste in company information ### What to be careful about You need to know **where what you type goes.** It differs by service. Be especially careful with: - Customer personal data (names, contact details, national ID, card numbers) - Contracts, unpublished financials - Passwords, API keys, connection details - Business plans not yet announced ### What to do about it **1. Check your company's policy first.** There may already be rules. **2. Strip what is not needed.** If you want a contract reviewed, the company name and the contact's name can go. The clause text alone is enough to review. ``` (as-is) "Hanbit Industries Co., represented by Kim Young-su..." (stripped) "Party A..." ``` **3. If it is for work, use a work environment.** A personal account and an environment your company has contracted for handle data differently. Some cases need an air-gapped setup. **4. Check whether your conversations are used for training, and turn it off if you can.** Depending on the service, what you type **may be used to train the model further.** Most have a setting for it somewhere. ``` In settings, look for a name like: "Help improve the model" "Data controls" "Use for training" ``` **The default differs by service and by plan.** Personal plans tend to have it on; an environment your company has contracted for tends to have it off. So "our company has a contract, so we are fine" and "my personal account must be the same" are two different statements. > **Once, right after you sign up.** Turning it off usually does not reach back > to conversations you already sent, so the order matters — **turn it off, then > paste in company material.** --- ## So when do you verify? It comes down to one question. > **What happens if this is wrong?** ```mermaid graph TD A["You got an answer"] --> B{"What if it's wrong?"} B -->|"just redo it"| C["Use it as-is
polishing · ideas · summaries"] B -->|"money, trust, or law"| D["Always verify
numbers · external docs · regulations"] ``` - The wording in an email draft is awkward → fix it. **Just use it.** - An amount going to a customer is wrong → that is an incident. **Always verify.** This same test is used at the organisational level later — it is exactly how you decide where to put human approval. ## Common questions ### "If I can't trust it, why use it?" Separate **producing a draft** from **final judgement.** AI takes the first, a person takes the second. Editing a draft is far faster than starting from a blank page, and that time saved is larger than the cost of checking. ### "Checking takes longer than doing it myself." Then **that task is not one to use it for.** You do not have to use it for everything. If verification costs more than writing, write it yourself. This judgement matters because the same mistake repeats at organisational scale — the third item in [how transformations fail](/guide/ax-failure-modes) is exactly this case. --- ## Check yourself **1. AI answered "our revenue last year was $4.7 million." What is wrong?**
Answer AI **cannot know your company's revenue.** Instead of saying so it produced a plausible number. That is hallucination, and because a wrong answer carries the same confidence as a right one, tone gives you no signal.
**2. What separates safe use from risky use?**
Answer **Processing material you supplied is safe** (summarising, translating, polishing); **filling in information AI does not have is risky** (numbers, sources, factual claims).
**3. What single question tells you whether verification is needed?**
Answer **"What happens if this is wrong?"** If you can just redo it, use it. If money, trust, or law is on the line, always verify.
--- ## That is individual use You have finished “New to AI”. You should now be in this state: - You know what AI is and how it differs from search - You know what in your work you could use it for - You have used it yourself and know how to get the answer you wanted - You know how far to trust it **From the next part the story moves from the individual to the organisation.** One person using it well and a company changing how it works are different problems, and most organisations stall in between → [Adoption is not transformation](/guide/ax-adoption-vs-transformation)