Runbook Craft

Parent: Writing and Documentation · researched 2026-05-29T18:25:26.688Z· 13 sources · 10 concepts · skill runbook-craft

A runbook is not a piece of documentation. It is a procedural script someone must execute correctly while tired, under pressure, with paging alerts firing in the background. Every step must be unambig

Overview

A runbook is not a piece of documentation. It is a procedural script someone must execute correctly while tired, under pressure, with paging alerts firing in the background. Every step must be unambiguous to a person who did not write it and may never have run it before. [source]

1. The "fresh machine" test

A runbook is only correct if a person who has never run it before, on a freshly provisioned environment, with no tribal context, can complete it successfully. [source]
Schedule a quarterly drill where someone who did not author the runbook runs it end-to-end on a sandbox or staging clone. Every pause is a defect in the runbook, not in the runner. [source]

2. Atomic, numbered steps with a verb-first imperative

Each step performs exactly one action that produces exactly one verifiable result. The verb comes first. [source]
Bad: "Now we need to make sure that the broker is running and you may also want to check the lag, and if the lag is high then restart things." [source]
Run kafka-broker-api status --broker mdb-prod-1. Expected output: STATUS: HEALTHY. [source]
Run kafka-consumer-groups --describe --group mdb-tam-consumer. Record the LAG column. [source]
If LAG > 50000, go to step 7 (broker restart). Otherwise continue to step 4. [source]

3. "You are here" markers and progress anchoring

Section banners at the top of each major phase: === PHASE 2 of 5: failover the primary ===. [source]
State-check steps at the boundary of each phase. [source]
Numbered top-level steps that never restart: use 1-25 across the whole runbook. [source]

4. Prerequisites block at the top, before step 1

A complete prerequisites block contains: [source]
Access: which SSO group, which IAM role, which secrets vault entry. [source]
Tools and versions: mongosh >= 2.0, aws-cli >= 2.13, jq. [source]
Inputs: cluster ID, account ID, ticket number. [source]
Approvals: who must sign off in writing before step 1. [source]
Communication: which Slack channel to post in. [source]

5. Rollback as a first-class section, defined before the change

A rollback section answers four questions: [source]
What signals trigger a rollback? Quantitative thresholds. ("Error rate > 2% sustained for 5 minutes." Not "if things look bad.") [source]
What is the rollback command? Exact, copy-pasteable. [source]
What is the rollback verification? [source]
What is the data-loss / state-loss implication? [source]

6. Decision points with measurable thresholds

Bad: "If memory looks high, restart the service." [source]
Good: "If mem_used_pct > 85 for 3 consecutive samples, restart the service (step 12)." [source]

7. Post-condition checks at the end of each phase

A post-condition check has three parts: [source]
The command to run (or signal to observe). [source]
The expected result (exact string, numeric range). [source]
What to do if the result does not match (rollback, escalate, retry). [source]

8. Ownership, review cadence, and metadata

Owner (a team, not a person). [source]
Last reviewed (date + name). [source]
Next review due (a real calendar date). [source]
Linked alert / page / dashboard. [source]
Estimated duration. [source]
Risk level (read-only / mutates non-prod / mutates prod / irreversible). [source]

9. Plain copy-pasteable commands, no placeholders in prose

Code-fenced blocks for every command, with no surrounding prose inside the block. [source]
A "set variables" step at the top that declares all substitutions once using environment variables. [source]

10. Common anti-patterns

The narrative blob: paragraphs where steps should be. [source]
Hardcoded secrets in code blocks. [source]
"You should know" gaps: the runbook assumes the runner has the same context as the author. [source]
Ambiguous phrasing: "investigate the issue", "check the dashboard." [source]
Outdated commands. [source]
No rollback. [source]

Full runbook skeleton

export CLUSTER_ID=<from alert> [source]

Decision Heuristics

When to split a runbook: more than 3 levels of branching, more than ~40 atomic steps, or two different audiences. [source]
When to automate vs. document: a runbook executed > 1x/month and fully deterministic is automation-eligible. [source]
When to mark a step "stop and escalate": any condition the runbook author did not anticipate, any post-condition mismatch. [source]
When to retire a runbook: the underlying alert hasn't fired in 12 months and the system has changed. [source]

References

Google SRE Workbook - On-Call [source]
PagerDuty Runbook Automation [source]
Nobl9 - Runbook Example: A Best Practices Guide [source]
Rootly - Incident Response Runbooks [source]

Children

The fresh-machine test (frontier)
Atomic numbered steps with verb-first imperatives (frontier)
You-are-here markers and progress anchoring (frontier)
Prerequisites block at the top (frontier)
Rollback as a first-class section (frontier)
Decision points with measurable thresholds (frontier)
Post-condition checks (assertions) (frontier)
Ownership, review cadence, and metadata (frontier)
Copy-pasteable commands and no placeholders in prose (frontier)
Common runbook anti-patterns (frontier)

Frontier under this node: Atomic numbered steps with verb-first imperatives, Common runbook anti-patterns, Copy-pasteable commands and no placeholders in prose, Decision points with measurable thresholds, Ownership, review cadence, and metadata, Post-condition checks (assertions), Prerequisites block at the top, Rollback as a first-class section, The fresh-machine test, You-are-here markers and progress anchoring

← the whole tree · 3D view· how to read this page