# Process 3 — automatic validation

> **What you will learn**
> How the self-healing loop runs, and the dangerous case where the loop
> manufactures a "pass."

## Tests are the only judge

The tests defined in [IntentOps](/guide/ha-intentops) become the judge here.

> **All pass means done. One failure means not done.**

There is no "almost there" or "good enough." That clarity is what makes
automation possible — you can move to the next stage without a person judging.

## The self-healing loop

On failure it does not call a person — **it fixes itself.**

```mermaid
graph TD
  T["Run the tests"] --> F{"All pass?"}
  F -->|"no"| D["Diagnose the failure"]
  D --> R["Regenerate the code"]
  R --> T
  F -->|"yes"| N["To deployment"]
```

One cycle looks like this:

```
Turn 1: test 3 fails — "expired coupon message differs"
        → diagnosis: wording does not match the specification
        → fix and rerun

Turn 2: test 5 fails — "total does not update"
        → diagnosis: missing state update
        → fix and rerun

Turn 3: all pass → next stage
```

**The human is not in this loop.** Rather than turning the loop, people look at
**whether the loop's criterion is correct.**

## This is where it most often collapses

The greatest weakness of this methodology.

> **When tests are shallow, the loop manufactures a "pass."**

An example of what that means:

```
Test: "after applying a coupon, the total must be less than the original"

Passing code AI produced:
  total = original - 1

→ The test passes. The intent is completely violated.
```

An extreme example, but the principle is the same. **The loop's goal is passing
tests, not satisfying intent.** When those diverge, the loop goes towards the
tests.

### Why this is dangerous

| | When a person builds | Self-healing loop |
|---|---|---|
| If tests are shallow | the person fills the gap with common sense | **it finds the gap precisely** |
| Result | generally as intended | satisfies the tests only |

A person knows "that's not it." The loop does not. **That is why shallow tests
are more dangerous with AI than with people.**

## So how do you use tests

### 1. Treat tests as the specification

```
✗ the code doesn't pass, so edit the test     ← the defence line is gone
✓ the code doesn't pass, so fix the code
✓ the test differs from the intent, so fix the specification first
```

**The moment you fit tests to the code**, the premise of this methodology
collapses.

### 2. Layer them

One layer cannot distinguish passing from correct.

| Layer | What it catches |
|---|---|
| Unit | the behaviour of one function |
| Integration | whether the parts work together |
| E2E | whether it actually works from the user's view |

With E2E, something like "total = original − 1" gets caught, because it is
visibly wrong on the actual screen.

### 3. Run it in a sandbox

Problematic code runs somewhere isolated **before it reaches production.** While
the loop is turning it must not touch real data.

### 4. Bound the loop

```
□ maximum retries (e.g. 5)
□ call a person when exceeded
```

Turning forever only burns cost. If five attempts do not fix it, **the problem is
in the specification or the tests** and a person needs to look.

## When you are evaluating AI output itself

Everything above is tests for **code.** If what you are building is the AI
response itself (customer support wording and so on), you use
[evals](/guide/ai-evaluation), not tests. Do not confuse the two.

---

## Check yourself

**1. What does the human do in a self-healing loop?**

<details>
<summary>Answer</summary>

**Not turn the loop, but check that the loop's criterion is correct.** Whether
the tests properly express the intent is the human's job.
</details>

**2. Why are shallow tests more dangerous with AI than with people?**

<details>
<summary>Answer</summary>

A person **fills gaps with common sense**, but the loop's goal is passing tests,
so it **finds the gaps precisely.** You get code like "total = original − 1"
which passes while violating the intent.
</details>

**3. Why must you not edit the test to make it pass?**

<details>
<summary>Answer</summary>

**Because the test is the only line of defence.** The moment you fit tests to
code, the premise that "tests judge done" collapses, and then
[regenerating code](/guide/ha-fluid-software) can no longer be verified.
</details>

---

Stand up gates on one of your own outputs. Forty minutes →
[Design your validation gates](/guide/ha-try-validation)