Build one pipeline end to end

What you will learn Not a chapter to read but one to do. Build a three-stage pipeline, then break the middle stage on purpose and see what the later stages do.

What you need

text

□ 60 minutes□ Schedules working□ Auto-start enabled — schedules do not fire if it is down□ One working directory that is easy to restore□ Do not include a stage that goes outside this time

If you have never set one up, read scheduled runs first.

Keep that last line. Put a send or a post in your first pipeline and step 4, where you break it deliberately, becomes a real incident.

Step 1 — Draw the three stages (10 min)

Draw it on paper first. The key is writing down what each stage leaves behind.

text

[1 collect]  ->  what it leaves:[2 shape]    ->  what it leaves:[3 summarise] -> what it leaves:

text

Example:  1 collect    every day at 09:00, write yesterday's changed files to a file  2 shape      read that list, group by type, write a table  3 summarise  read the table, write a three-line summary

Every stage must leave a file. A stage that holds its result only in its head cannot be re-run from the middle — this is what "stages must be re-runnable" in pipeline design actually looks like.

Step 2 — Build stage 1 alone and run it (15 min)

Do not build all three at once. Stand one up and verify it.

text

□ You created the stage 1 schedule□ You ran it once by hand□ The output file actually appeared□ You opened that file and checked the contents

Do not skip the fourth. A file existing and a file being right are different things. Skip it and, when stage 3 comes out wrong, you cannot tell where it went wrong.

Step 3 — Chain stages 2 and 3 (15 min)

Build each later stage so that it reads the previous stage's output.

text

□ Stage 2 reads stage 1's output file□ Stage 3 reads stage 2's output file□ Each stage runs later than the one before   <- check the gaps

How to set the gaps

text

Stage 1 09:00  ->  Stage 2 09:20  ->  Stage 3 09:40

Do not put stage 2 at 09:05 because stage 1 usually takes three minutes. There will be slow days. Leave at least three times the usual duration, or write the stage so it stops when the previous output is missing.

text

The line to put in stages 2 and 3:  "If there is no file to read, produce nothing and leave only 'no previous result'."

Step 4 — Break the middle (15 min)

This is the point of the exercise. Rename or delete stage 1's output file, then run stages 2 and 3.

What to look for

What stage 2 did	Verdict
Left "no previous result" and stopped	✓ properly designed
Produced an empty table and completed	stage 3 now summarises that empty table
Found and used an older file	the worst — today's report goes out on yesterday's data

That third row is the classic production incident. No error is raised, so nobody notices. The numbers just quietly slip by a day.

Check all the way to stage 3

text

□ You opened stage 3's output□ Did "none" propagate into it, or did you get a plausible summary?

If it did not propagate, fix the stage 2 instruction and do step 4 again. One stage's failure being laundered into success by the next is the most common way a pipeline fails.

Step 5 — Cost and cleanup (5 min)

text

□ You counted how many times a day each of the three runs□ You turned off the practice schedules□ For anything you kept, the failure behaviour is in the instruction

Do count the first one. Every stage carries cost, and running daily makes it accumulate. Three stages every day is ninety runs a month.

Self-check

text

□ Every stage leaves a file□ You stood the stages up one at a time and verified each□ Gaps are at least three times the usual duration□ You broke the middle and checked the later stages   <- skip this and you did not do it□ "No previous result" propagated all the way to stage 3

Check yourself

1. Why must every stage leave a file?

Answer

So you can re-run from the middle. Without the result on disk you cannot re-run stage 2 alone; you have to redo everything from the start.

2. Why is "found and used an older file" the most dangerous outcome?

Answer

No error is raised, so nobody notices. The pipeline reports success and today's report goes out built on yesterday's data.

3. Why leave generous gaps between stages?

Answer

There will always be slow days. If the next stage runs before the previous finished, it proceeds on a missing or stale result. Widen the gap, or write the stage to stop when there is no result.

Next, turning the repeated procedure itself into a reusable unit → Capturing procedures as skills