# Evaluation (evals)
> **What you will learn**
> An individual can judge by eye; an organisation cannot. What to measure, how,
> and what happens without it.
## The difference between individual and organisation
An individual needs no evaluation. Look at the answer, and if you do not like it,
say so again. **Judgement happens instantly and for free.**
An organisation is different.
| | Individual | Organisation |
|---|---|---|
| Who judges | you | a different person each time |
| When | immediately | later, or never |
| Standard | your instinct | **different per person** |
| Volume per day | a few | hundreds to thousands |
If people look at hundreds of items, the gain from automating disappears. **So
judgement itself has to be automated, and that is evaluation.**
## What happens without it
```mermaid
graph TD
A["Edit the prompt"] --> B["Seems better"]
B --> C["Deploy"]
C --> D["A different case gets worse"]
D --> E["Nobody notices"]
E --> A
```
**The most common symptom**: "I edited the prompt, this case got better but that
one got worse." Except nobody looks at that one, so nobody knows. A few
iterations of this and **nobody can say which version is best.**
Software already went through this problem, and the answer is the same —
**regression tests.**
## What you build — a golden dataset
The core is **a set of examples with correct answers attached.**
```
Case 1
Input: "When's my order from yesterday arriving? Order A-12345"
Expected: category=delivery_status, order_number=A-12345
Case 2
Input: "Can I get a refund on this"
Expected: category=refund, order_number=none → must ask back
Case 3
Input: "It was late last time and it's late again, seriously"
Expected: category=complaint, sentiment=negative, escalate to a person
...
```
### How many do you need?
**Twenty to thirty is enough to start.** You do not need thousands.
What matters is not the count but **what is in it.**
| Must include | Why |
|---|---|
| The most common cases | they determine overall quality |
| **Cases that actually went wrong** | prevents the same mistake recurring |
| Ambiguous edge cases | where judgement diverges |
| Cases where it must refuse | e.g. it should refuse requests for personal data |
**The second is the key one.** When something goes wrong in production, add it
to the dataset. Then the dataset improves over time.
## How to score
Three approaches, usually mixed.
### 1. Exact match (most reliable)
For classification or extraction, where **there is one right answer.**
```
expected: category=delivery actual: category=delivery → pass
expected: order_number=A-12345 actual: order_number=A-1234 → fail
```
Build it this way when you can. **There is nothing to argue about in scoring.**
### 2. Rule checks
Where there is not one right answer but there are **conditions to satisfy.**
```
□ is the reply three sentences or fewer
□ if an amount is mentioned, does it match the lookup result
□ did it avoid asking for personal data
□ did it follow the required format (JSON)
```
### 3. Model scoring (LLM-as-judge)
For things you cannot write as rules — tone, appropriateness — have another
model score it.
```
Evaluate the customer support reply below.
Criteria: is it polite / did it actually answer the question / did it
avoid inventing information it did not have.
Answer pass or fail for each criterion only.
```
> **Caution**: the scoring model is also wrong sometimes. So **a person must spot
> check the scoring results occasionally.** Without that, nobody knows when the
> scorer is wrong.
## When to run it
```
□ every time you edit a prompt ← the most important
□ when you change models
□ when you add or change tools
□ periodically (weekly)
```
**The first line is the key one.** Edit a prompt without running evals and you
are back in the diagram above.
## How to start in practice
There is no need to start elaborately.
```
Week 1: gather 20 real cases in a spreadsheet
(two columns — input and expected — is enough to start)
Week 2: run all 20 and count how many pass → this is your baseline
Week 3: rerun all 20 every time you edit the prompt
After: when a case goes wrong, add it to the dataset
```
**Week 2's baseline matters.** If you do not know your current score, you do not
know whether it improved. Same principle as
[recording the current value in Phase 1](/guide/ax-phase1-problem).
## Connection to hyper-agile
[Hyper-agile](/guide/ha-micro-sprint) says "tests are the only judge of done."
For code that test is a unit test; **for AI output it is this evaluation.**
They are different things.
| | Subject | Judgement |
|---|---|---|
| Unit test | code behaviour | pass/fail is unambiguous |
| **Evals** | **AI output** | **statistical. Something like 90% pass** |
AI output differs slightly every time, so "100% pass" is not the goal.
**"Better than the baseline"** is the judgement.
## Common misconceptions
### "We're too small to need this."
Size is irrelevant. A spreadsheet with 20 cases is an eval. Without one, every
prompt edit becomes **a judgement by instinct**, and that judgement is often
wrong.
### "Isn't building the evals more work than the thing itself?"
Gathering 20 cases takes half a day. Without that half day you can **never
answer "is this version better?"** — and that question comes up dozens of times
ahead of you.
---
## Check yourself
**1. Why does an individual not need evals but an organisation does?**
Answer
An individual judges **instantly and for free**, while in an organisation the
judge differs each time, the standard differs, and the volume is hundreds to
thousands. If people review it all, the gain from automating disappears — so
judgement itself must be automated.
**2. Which of the must-includes in a golden dataset matters most?**
Answer
**Cases that actually went wrong.** Adding production errors to the dataset stops
the same mistake recurring, and the dataset improves over time.
**3. How do evals differ from unit tests?**
Answer
Unit tests are unambiguously pass or fail, while **AI output varies slightly
every time, so it is viewed statistically.** The standard is not "100% pass" but
"better than the baseline."
---
Next: what to pick, and what it costs →
[Choosing a model, and cost](/guide/ai-choosing-models)