A runbook is a procedure with steps, written once and run many times. It is the difference between an agent that can answer a question about a refund and one that can process it.
Where the approval step goes
Section titled “Where the approval step goes”Runbooks can pause and wait for a person. Use that for anything with a real consequence: refunds, cancellations, anything that writes to a system of record, anything a customer would be upset to have happen twice.
An approval costs a few seconds. Not having one costs an afternoon, and occasionally a customer.
The test is simple. If a step went wrong at three in the morning, could you undo it by lunchtime? If not, put a person in front of it.
Writing one that survives contact with reality
Section titled “Writing one that survives contact with reality”Make steps small. A step that does four things fails in four ways and tells you about one of them.
Make failure informative. A step that stops with “error” leaves the person who picks it up doing forensics. A step that stops with “the payment provider returned insufficient funds” leaves them doing their job.
Assume it will be interrupted. Deploys happen, connections drop. A runbook that cannot be resumed will eventually be a runbook someone finishes by hand while reading logs.
Automations, tasks and events
Section titled “Automations, tasks and events”These get conflated, so plainly:
| Is | |
|---|---|
| Event | Something happened |
| Automation | When this event happens, do that |
| Task | A piece of work tracked to done, often raised by an automation |
| Runbook | A procedure with steps, run deliberately or by an automation |
An automation is a reflex. A runbook is a plan. Reach for the reflex when the response is always the same, and the plan when it has steps a person might need to watch.
Webhooks
Section titled “Webhooks”Webhooks call your systems when something happens here. Two things save pain later: return quickly and do the work asynchronously, and make your handler idempotent, because at-least-once delivery means the same event will arrive twice on the day you least want it to.