# Audit trail and observability

> **What you will learn**
> What an agent system has to record, why ordinary server logs are not enough,
> and what must never go in.

## If you can't answer, you haven't deployed

Some day after launch these questions arrive.

```
"Why did that email go out?"
"Where did this number come from?"
"Who approved it?"
"Why did cost double last month?"
```

**None of them can be answered without a record.** And a system that cannot
answer them gets switched off at the next review.

## Four reasons

| Reason | What it needs |
|---|---|
| **Regulation** | high-impact AI under [the AI Framework Act](/guide/ax-compliance) requires documentation retained **5 years** |
| **Incident investigation** | which document did a rogue instruction arrive in |
| **Improvement** | where does it fail most, why does rework happen |
| **Cost** | which task is consuming the usage |

**The first row is widely misread.** It is not enough to have taken the
measures — you need **documentation confirming them**, kept five years. That
cannot be produced after the fact.

## Ordinary logs are not enough

Traditional server logs are **one request, one line**. Agents are not.

```
One user request
  ├ model call (make a plan)
  ├ tool: file_search      → 12 results
  ├ model call (decide which files to open)
  ├ tool: read_file ×3
  ├ model call (draft)
  ├ waiting on human approval  → approved
  └ tool: send_email       → done
```

**One line cannot show that structure.** What you need is not a log but a
**trace** — a tree. "Why did it do that?" is only answerable by walking that
tree.

The industry is converging here on **OpenTelemetry's GenAI semantic
conventions** — a standard set of attributes for model calls, agent runs, tool
executions, and session metrics. As of 2026 the major coding agents emit records
in this format. **Look at the standard before building your own**; it keeps your
records usable when you change tooling later.

## What to record

```
□ Who asked for what       user, workspace, the request as written
□ What it used as basis    documents referenced, sources of retrieved results
□ Which tools, how called  tool name, arguments, result summary, duration
□ What a human decided     approve/deny, who, when, reason for denial
□ What left the building   outbound actions and their destinations
□ What it cost             tokens per model, call counts
□ How it ended             success/failure/aborted, error detail
```

**Note the fourth row.** Approval records are the first thing asked for in both
regulatory response and incident investigation — and they are frequently shown
on screen but never stored.

## What never to record

Thinking only about recording more is where the accident happens.

| Never | Why |
|---|---|
| API keys, tokens, passwords | logs have wide read access. The top leak path |
| Personal data inside raw prompts | needs a legal basis and a retention period |
| Full customer documents | keep a summary and an identifier; leave the original where it lives |

> **Retention requirements conflict.** Regulation says "keep five years";
> data-protection law says "keep the minimum." The answer is **separation.**
> What audit needs (who, when, what was approved) for five years; the contents
> (raw text, personal data) briefly. Design two stores from day one.

## When to add it

```mermaid
graph TD
  A["Phase 3 — POC"] -->|"turn tracing on"| B["you can see where it fails"]
  B --> C["Phase 4 — rollout"]
  C -->|"design retention and access control"| D["operations"]
  D --> E["incident investigation · regulatory response · cost analysis"]
```

**Turn it on at POC.** Add it after launch and you cannot answer "but why did it
work in the POC?" Traces from the POC also surface the failure types that become
material for a quality-check case set (an eval dataset).

## In the product

| What is recorded | Where |
|---|---|
| Per-workspace usage and credit consumption | [usage and credits](/guide/cn-usage) |
| Tool calls and results kept on the thread | [threads](/guide/cn-thread) |
| Scheduled and queued run history | [queue and scheduling](/guide/cn-queue) |
| Per-system execution records | [managing systems](/guide/cn-systems-manage) |

## Common misconceptions

### "Does it record why the model decided something?"

**No.** What is recorded is **what it saw and what it did**, not internal
reasoning. So the answer to "why" is always a **reconstruction from inputs and
actions.** Fortunately the explanation duty in regulation usually asks for
exactly that level — what material, through what steps, to what conclusion.

### "Isn't storing all of that expensive?"

**It is if you store full text.** Keep tool results as a summary plus an
identifier pointing at the original and the volume drops to single-digit
percentages. As noted above, storing raw text is also a liability on the
personal-data side.

---

## Check yourself

**1. Why can't ordinary server logs trace an agent?**

<details>
<summary>Answer</summary>

**Because one request branches into many model calls and tool calls.** A
one-line log cannot show that structure. "Why did it do that?" is answered by
walking a tree-shaped trace.
</details>

**2. How do five-year retention and data minimisation coexist?**

<details>
<summary>Answer</summary>

**Split the stores.** Audit-relevant fields (who, when, what was approved) for
five years; contents (raw text, personal data) briefly. It has to be designed as
two stores from the start to be separable later.
</details>

**3. Why turn tracing on during the POC?**

<details>
<summary>Answer</summary>

**Because adding it after launch leaves "why did it work in the POC?"
unanswerable.** POC traces also reveal the failure types that become material
for the eval dataset.
</details>

---

Technology and process are in place. The last piece is people →
[Change management](/guide/ax-change-management)