Process 3 — automatic validation

What you will learn How the self-healing loop runs, and the dangerous case where the loop manufactures a "pass."

Tests are the only judge

The tests defined in IntentOps become the judge here.

All pass means done. One failure means not done.

There is no "almost there" or "good enough." That clarity is what makes automation possible — you can move to the next stage without a person judging.

The self-healing loop

On failure it does not call a person — it fixes itself.

graph TD
  T["Run the tests"] --> F{"All pass?"}
  F -->|"no"| D["Diagnose the failure"]
  D --> R["Regenerate the code"]
  R --> T
  F -->|"yes"| N["To deployment"]

One cycle looks like this:

text

Turn 1: test 3 fails — "expired coupon message differs"        → diagnosis: wording does not match the specification        → fix and rerunTurn 2: test 5 fails — "total does not update"        → diagnosis: missing state update        → fix and rerunTurn 3: all pass → next stage

The human is not in this loop. Rather than turning the loop, people look at whether the loop's criterion is correct.

This is where it most often collapses

The greatest weakness of this methodology.

When tests are shallow, the loop manufactures a "pass."

An example of what that means:

text

Test: "after applying a coupon, the total must be less than the original"Passing code AI produced:  total = original - 1→ The test passes. The intent is completely violated.

An extreme example, but the principle is the same. The loop's goal is passing tests, not satisfying intent. When those diverge, the loop goes towards the tests.

Why this is dangerous

	When a person builds	Self-healing loop
If tests are shallow	the person fills the gap with common sense	it finds the gap precisely
Result	generally as intended	satisfies the tests only

A person knows "that's not it." The loop does not. That is why shallow tests are more dangerous with AI than with people.

So how do you use tests

1. Treat tests as the specification

text

✗ the code doesn't pass, so edit the test     ← the defence line is gone✓ the code doesn't pass, so fix the code✓ the test differs from the intent, so fix the specification first

The moment you fit tests to the code, the premise of this methodology collapses.

2. Layer them

One layer cannot distinguish passing from correct.

Layer	What it catches
Unit	the behaviour of one function
Integration	whether the parts work together
E2E	whether it actually works from the user's view

With E2E, something like "total = original − 1" gets caught, because it is visibly wrong on the actual screen.

3. Run it in a sandbox

Problematic code runs somewhere isolated before it reaches production. While the loop is turning it must not touch real data.

4. Bound the loop

text

□ maximum retries (e.g. 5)□ call a person when exceeded

Turning forever only burns cost. If five attempts do not fix it, the problem is in the specification or the tests and a person needs to look.

When you are evaluating AI output itself

Everything above is tests for code. If what you are building is the AI response itself (customer support wording and so on), you use evals, not tests. Do not confuse the two.

Check yourself

1. What does the human do in a self-healing loop?

Answer

Not turn the loop, but check that the loop's criterion is correct. Whether the tests properly express the intent is the human's job.

2. Why are shallow tests more dangerous with AI than with people?

Answer

A person fills gaps with common sense, but the loop's goal is passing tests, so it finds the gaps precisely. You get code like "total = original − 1" which passes while violating the intent.

3. Why must you not edit the test to make it pass?

Answer

Because the test is the only line of defence. The moment you fit tests to code, the premise that "tests judge done" collapses, and then regenerating code can no longer be verified.

Stand up gates on one of your own outputs. Forty minutes → Design your validation gates