Why instructions do not simply stick
What you will learn This is the fourth property. Why being able to change behaviour with words is the very same reason what you meant and what landed drift apart, what that drift looks like, and what actually works.
It is not keeping the rule, it is continuing it
It did not break a rule. The model does not store your rule somewhere and check each answer against it. It produces the natural continuation of "a conversation with a no-tables sentence further up." The further away that sentence gets — and the more table-shaped the content is — the more the continuation tips back toward a table.
An instruction is not a switch. It is a weight. It does not turn something on or off; it tilts a probability.
That is steerability. Being able to change role, format, and tone with a few lines is the capability side of this property; those changes being nothing more than probability is the limit side. They are the same fact.
Five shapes the drift takes
1. Distance dilutes it
The longer the conversation, the fainter the rule you gave at the start. When it collides with the third property (working memory) the rule can vanish outright — but it is already weakening well before it falls off. Still inside the window, but far away, means weak.
2. Negatives summon the thing
"Do not do X" is an act of putting X into the conversation. The moment you write the forbidden word, it becomes context. The same is true of people, but it shows up more often with models.
- ✗ "Do not use the customer's name" → answers containing the name go up
- ✓ "Refer to the customer only as 'Company A'" → you gave it a slot to fill
The principle is give a substitute, not a prohibition.
3. Rules collide with each other
Those two cannot both hold in one answer. The model will not tell you they conflict; it quietly drops one. Usually whichever came later, and whichever is more specific, wins.
When an answer looks wrong, the first thing to read is not the model but your list of instructions. Look for two lines fighting.
4. It keeps the format and drops the content
This is where people are fooled most often.
Format instructions land far more reliably than content instructions. Format is surface, so it is easy to continue; "ground everything in the material" requires a judgement at every sentence. So a perfectly formatted answer tricks you into believing the content was obeyed too.
5. Data gets read as instruction
If the material you pasted contains an imperative sentence, the model may read it as an instruction, because to the model "instruction" and "material" are the same characters.
When that property is turned into an attack, it is prompt injection → When what it reads becomes a command
What actually works
| Method | Effect | Why |
|---|---|---|
| Restating the rule | Large | Closer means heavier |
| Giving a substitute instead of a ban | Large | It gives the slot something to hold |
| Attaching one or two examples | Large | An example is more precise than an adjective |
| Marking rules and material apart | Large | Reduces material being read as instruction |
| Making it recite the rules back | Medium | At least you can check compliance |
| Adding "always", "never" | Small | Slightly more weight; same property |
| Switching to a better model | Small | Frequency only |
Marking rules and material apart
Marking is not a guarantee. It tilts a probability. So on genuinely dangerous paths you put structure rather than wording — narrow the tool permissions and insert a human approval (Tools are permissions).
A cheap way to check compliance
It is the model grading itself, so it is not complete. Still, it catches a good share of kept the format, broke the content cases. For repeated work, automating that check is what evaluation (evals) is.
Where this lives in an organisation
In one person's chat window you can retype the rules every time. In automation used by many people you cannot — the rules have to live in the system, not in someone's habit.
In Connect that place is a sub-agent's instructions. In HyperTeams it is skills.
Check
1. What does "an instruction is a weight, not a switch" mean?
Answer
The model does not store a rule and check against it; it produces the continuation of a text containing that rule. So an instruction tilts a probability rather than switching behaviour. It fades with distance, and loses when the opposite content is more natural.
2. Why does "do not use the customer's name" backfire, and what do you write instead?
Answer
Because writing the forbidden word puts it into the context. A negative gives nothing to fill the slot with. Give a substitute — "refer to the customer only as 'Company A'".
3. You see an answer with perfect formatting but wrong content. What should you suspect?
Answer
That format instructions land much more reliably than content instructions. The surface being obeyed does not mean "ground everything in the material" was obeyed. Make it recite compliance separately, or check with evals.
Now rewrite some instructions for real → Rewriting instructions