Recover a stalled job

What you will learn Not a chapter to read but one to do. Break a job deliberately and walk the recovery once — so that the first time is not the time it actually matters.

What you need

text

□ 30 minutes□ One registered working directory□ A directory that is easy to restore — commit first if you use git□ Knowing how to open the log

If you are not sure where the log lives, troubleshooting says where.

Keep that third line. This exercise interrupts things on purpose.

Step 1 — Give it something slow (5 min)

A job that finishes instantly leaves no room to interrupt. Ask for something slightly long-running.

text

Go through every file in this folder and write one line on what each one does.Do not modify any files.

text

□ Is there an in-progress indicator?□ Can you see which file it is reading?

Step 2 — Interrupt it (5 min)

While it runs, press stop.

What to look for

Check
Does the state change to stopped?	□
Is how far it got still recorded?	□
Can you see partial results?	□

The second one matters. Without a record of how far it got, this is not recovery — it is starting over.

Step 3 — Break it harder (10 min)

This time kill the program itself. Start the job again and shut it down mid-run.

text

□ Start the job□ A few seconds later, terminate the program (close the window or stop it)□ Start it again

What to check after restarting

text

□ Is the job still in the list?□ What state is shown (failed? interrupted? still in progress?)□ If it says "in progress" — that is a lie. Nothing is running

Do check the third line. When the program dies, whatever was holding that job dies with it, but the state on screen can still be whatever was last saved. Do not trust "in progress" on screen — read the log.

Step 4 — Write the recovery order (10 min)

Turn what you just did into an order someone else can follow.

text

1. Check the state   — where do you look:2. Check the log     — which file, last how many lines:3. Decide            — re-run, or use the partial result:4. Restart           — how:5. If it still fails — who do you tell, holding what:

For "holding what" in step 5, use the three version lines you wrote in check your first connection.

If anything needed undoing

text

□ Was it interrupted with files already changed?□ Did you undo the change?□ If you use git — did you confirm the undo actually works?

Confirm that last line now. A safety net should not be used for the first time in an emergency.

Self-check

text

□ You interrupted once with stop and once by killing the program□ You saw that the screen state and the real state can differ□ You wrote the five-line recovery order□ You confirmed that undoing actually works

Check yourself

1. Why not trust "in progress" on screen after the program has died?

Answer

Whatever was holding the job died with the program. The screen only shows the last saved state; nothing is actually working on it. Read the log.

2. Why does "how far it got" matter when you stop a job?

Answer

Without it, recovery is just starting over. With a partial result you can resume from there, or use just that part.

3. Why test undoing in advance?

Answer

A safety net should not be used for the first time in an emergency. If you have never confirmed the undo works, it becomes one more thing to debug at the worst moment.

To go beyond the screen and connect other programs → The two API surfaces