Building pipelines with schedules

What you will learn The two kinds of schedule and when to use each, and how to chain several stages.

There are two kinds

Scheduled runs covered only task schedules. There are actually two.

Kind	What it does	What it needs
`task`	hands an instruction to an agent	an instruction
`terminal`	runs a command as-is	a command

This distinction is the heart of pipeline design.

Which to use when

graph TD
  Q["Which is this stage?"] --> Q1{"Does it need judgement<br/>each time?"}
  Q1 -->|"yes"| T["task<br/>agent"]
  Q1 -->|"no"| Q2{"Does one command do it?"}
  Q2 -->|"yes"| S["terminal<br/>command"]
  Q2 -->|"no"| T

Stage	Kind	Why
Read material and summarise	`task`	content differs each time
Convert file formats	`terminal`	a fixed command
Write a script	`task`	needs judgement
Render or build	`terminal`	a fixed command
Analyse error logs	`task`	differs each time
Back up or clean up	`terminal`	a fixed command

Do not use task for a stage with no judgement. Calling an agent costs money and gives slightly different results each time. terminal is cheap and deterministic.

This is the same judgement as Phase 1's AI suitability assessment — if it is 100% expressible as rules, there is no reason to use AI.

Creating one schedule

What goes into a new schedule:

Item	`task`	`terminal`
Working directory	required	required
Cadence	required	required
Instruction	required	—
Command	—	required
Autonomy	optional	not applicable
Engine, model	optional	not applicable

The cadence is as in scheduled runs — minutes, hours, daily, weekdays, weekly, monthly, or a cron expression typed directly.

Chaining stages

This is the substance of the chapter. One schedule is one stage. A pipeline has to chain several.

Method 1 — stagger them (simplest)

text

06:00  terminal  collect material07:00  task      write the script from what was collected08:00  terminal  produce the output from the script

Upside: simple. Each stage is independent, so one failing leaves the others running. Downside: if an earlier stage runs late, the next one runs empty-handed.

So every stage has to verify its preconditions.

text

✓ "If the collection output file is missing, do nothing and report it"

What scheduled runs called "have it verify preconditions" becomes mandatory in a pipeline.

Method 2 — one stage triggers the next

Inside a terminal stage's command, create the next task via the REST API.

bash

# After collecting, create the next task only if there is a resultif [ -s out/collected.json ]; then  curl -X POST http://localhost:27777/api/v1/tasks \    -H "Authorization: Bearer $HT_KEY" \    -H "Content-Type: application/json" \    -d '{"workspaceId":3,"prompt":"Write the script from out/collected.json","name":"script-'"$(date +%F)"'"}'fi

Upside: the next stage runs only after the previous finishes. No empty runs. Downside: the key ends up in a script. Move it to an environment variable.

Method 3 — one task does everything

Put several stages in one instruction and let the agent work through them.

Generally not recommended. The context fills so consistency drops towards the end, and it is hard to tell which stage failed. The micro-sprint principle applies — split them and each starts with clean context.

Design principles

1. Leave a file at every stage

text

out/  01-collected.json  02-script.md  03-audio/  04-final.mp4

With intermediate outputs on disk you can rerun from the failed stage. Pass them only in memory and you start over.

The numbers exist so the order is visible.

2. Skip what already exists

text

✓ "Skip any paragraph that already has a file in out/03-audio/"

When 3 of 50 fail, you regenerate 3.

3. Make it report failures

The most frequently omitted. Fail quietly and you find out weeks later.

text

✓ "Report success or failure at the end of each stage.   On failure, do not proceed to the next stage."

4. Start with low autonomy

Pipelines run when nobody is watching. Do not set autonomy to fully automatic from the start.

text

Week 1: run as plan only and read the plan dailyWeek 2: auto-approve file editsAfter:  raise it once stable

Cost accumulates

A daily pipeline costs one run × 30.

text

3 task stages × $0.30 × 30 days = $27/month

The order for reducing it is fixed.

text

1. Move judgement-free stages to terminal   ← the biggest2. Lighter models for simple tasks3. Cap failure retries

The first is the most effective. If format conversion was being handed to an agent, turning it into one command takes that stage's cost to zero.

Managing them

As schedules multiply they need managing.

text

□ when did this schedule last succeed□ is a pipeline whose purpose ended still running□ is a later stage still running while an earlier one fails

The third is specific to pipelines. Collection may have been failing for weeks while script writing keeps running and producing empty results.

Check yourself

1. How do you choose between task and terminal?

Answer

Judgement each time means task; a fixed command means terminal. Using task for a judgement-free stage costs money and gives varying results.

2. Why leave intermediate outputs as files?

Answer

So you can rerun from the failed stage. Without files on disk, a failure at any stage means starting over.

3. What most effectively reduces pipeline cost?

Answer

Moving judgement-free stages to terminal. Turning something like format conversion into one command removes that stage's cost.

Build a three-stage one and break the middle. Sixty minutes → Build one pipeline end to end