Runbook Craft
Parent: Writing and Documentation · Published reference · snapshot 2026-09-08 · skill technical-writing-craft/references/runbook-craft.md
↓ Facts as markdownall context files
A runbook is not a piece of documentation. It is a procedural script someone must execute correctly while tired, under pressure, with paging alerts firing in the background. Every step must be unambig
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Overview
- A runbook is not a piece of documentation. It is a procedural script someone must execute correctly while tired, under pressure, with paging alerts firing in the background. Every step must be unambiguous to a person who did not write it and may never have run it before. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#overview]
1. The "fresh machine" test
- A runbook is only correct if a person who has never run it before, on a freshly provisioned environment, with no tribal context, can complete it successfully. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#1-the-fresh-machine-test]
- Schedule a quarterly drill where someone who did not author the runbook runs it end-to-end on a sandbox or staging clone. Every pause is a defect in the runbook, not in the runner. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#1-the-fresh-machine-test]
2. Atomic, numbered steps with a verb-first imperative
- Each step performs exactly one action that produces exactly one verifiable result. The verb comes first. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#2-atomic-numbered-steps-with-a-verb-first-imperative]
- Bad: "Now we need to make sure that the broker is running and you may also want to check the lag, and if the lag is high then restart things." [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#2-atomic-numbered-steps-with-a-verb-first-imperative]
- Run kafka-broker-api status --broker mdb-prod-1. Expected output: STATUS: HEALTHY. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#2-atomic-numbered-steps-with-a-verb-first-imperative]
- Run kafka-consumer-groups --describe --group mdb-tam-consumer. Record the LAG column. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#2-atomic-numbered-steps-with-a-verb-first-imperative]
- If LAG > 50000, go to step 7 (broker restart). Otherwise continue to step 4. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#2-atomic-numbered-steps-with-a-verb-first-imperative]
3. "You are here" markers and progress anchoring
- Section banners at the top of each major phase: === PHASE 2 of 5: failover the primary ===. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#3-you-are-here-markers-and-progress-anchoring]
- State-check steps at the boundary of each phase. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#3-you-are-here-markers-and-progress-anchoring]
- Numbered top-level steps that never restart: use 1-25 across the whole runbook. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#3-you-are-here-markers-and-progress-anchoring]
4. Prerequisites block at the top, before step 1
- A complete prerequisites block contains: [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#4-prerequisites-block-at-the-top-before-step-1]
- Access: which SSO group, which IAM role, which secrets vault entry. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#4-prerequisites-block-at-the-top-before-step-1]
- Tools and versions: mongosh >= 2.0, aws-cli >= 2.13, jq. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#4-prerequisites-block-at-the-top-before-step-1]
- Inputs: cluster ID, account ID, ticket number. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#4-prerequisites-block-at-the-top-before-step-1]
- Approvals: who must sign off in writing before step 1. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#4-prerequisites-block-at-the-top-before-step-1]
- Communication: which Slack channel to post in. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#4-prerequisites-block-at-the-top-before-step-1]
5. Rollback as a first-class section, defined before the change
- A rollback section answers four questions: [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#5-rollback-as-a-first-class-section-defined-before-the-change]
- What signals trigger a rollback? Quantitative thresholds. ("Error rate > 2% sustained for 5 minutes." Not "if things look bad.") [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#5-rollback-as-a-first-class-section-defined-before-the-change]
- What is the rollback command? Exact, copy-pasteable. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#5-rollback-as-a-first-class-section-defined-before-the-change]
- What is the rollback verification? [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#5-rollback-as-a-first-class-section-defined-before-the-change]
- What is the data-loss / state-loss implication? [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#5-rollback-as-a-first-class-section-defined-before-the-change]
6. Decision points with measurable thresholds
- Bad: "If memory looks high, restart the service." Good: "If mem_used_pct > 85 for 3 consecutive samples, restart the service (step 12)." [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#6-decision-points-with-measurable-thresholds]
7. Post-condition checks at the end of each phase
- A post-condition check has three parts: [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#7-post-condition-checks-at-the-end-of-each-phase]
- The command to run (or signal to observe). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#7-post-condition-checks-at-the-end-of-each-phase]
- The expected result (exact string, numeric range). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#7-post-condition-checks-at-the-end-of-each-phase]
- What to do if the result does not match (rollback, escalate, retry). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#7-post-condition-checks-at-the-end-of-each-phase]
8. Ownership, review cadence, and metadata
- Owner (a team, not a person). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#8-ownership-review-cadence-and-metadata]
- Last reviewed (date + name). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#8-ownership-review-cadence-and-metadata]
- Next review due (a real calendar date). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#8-ownership-review-cadence-and-metadata]
- Linked alert / page / dashboard. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#8-ownership-review-cadence-and-metadata]
- Estimated duration. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#8-ownership-review-cadence-and-metadata]
- Risk level (read-only / mutates non-prod / mutates prod / irreversible). [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#8-ownership-review-cadence-and-metadata]
9. Plain copy-pasteable commands, no placeholders in prose
- Code-fenced blocks for every command, with no surrounding prose inside the block. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#9-plain-copy-pasteable-commands-no-placeholders-in-prose]
- A "set variables" step at the top that declares all substitutions once using environment variables. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#9-plain-copy-pasteable-commands-no-placeholders-in-prose]
10. Common anti-patterns
- The narrative blob: paragraphs where steps should be. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#10-common-anti-patterns]
- Hardcoded secrets in code blocks. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#10-common-anti-patterns]
- "You should know" gaps: the runbook assumes the runner has the same context as the author. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#10-common-anti-patterns]
- Ambiguous phrasing: "investigate the issue", "check the dashboard." [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#10-common-anti-patterns]
- Outdated commands. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#10-common-anti-patterns]
Full runbook skeleton
- export CLUSTER_ID=<from alert> [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#full-runbook-skeleton]
Decision Heuristics
- When to split a runbook: more than 3 levels of branching, more than ~40 atomic steps, or two different audiences. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#decision-heuristics]
- When to automate vs. document: a runbook executed > 1x/month and fully deterministic is automation-eligible. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#decision-heuristics]
- When to mark a step "stop and escalate": any condition the runbook author did not anticipate, any post-condition mismatch. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#decision-heuristics]
- When to retire a runbook: the underlying alert hasn't fired in 12 months and the system has changed. [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#decision-heuristics]
References
- Google SRE Workbook - On-Call [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#references]
- PagerDuty Runbook Automation [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#references]
- Nobl9 - Runbook Example: A Best Practices Guide [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#references]
- Rootly - Incident Response Runbooks [archived source: https://llms-explorer.com/sources/mdb-context-hub/runbook-craft/#references]
Where this helps
- Writing an on-call procedure that someone under paging pressure at 3am can execute correctly without first needing to understand the underlying system deeply. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Documenting a rollback path before the risky change ships, so recovery is a rehearsed, first-class procedure rather than something improvised during an incident. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Defining decision points with measurable thresholds, not vague judgment calls, so two different responders make the same call given the same data. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Passing the "fresh machine" test — verifying a runbook works when followed literally on a machine with none of the author's assumed local state or context. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Establishing ownership and a review cadence for a runbook so it doesn't silently go stale as the underlying system changes. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
How to apply this
- Write every step as an atomic, numbered, verb-first imperative instead of descriptive prose that requires interpretation under pressure. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Put the prerequisites block at the very top, before step 1, so a responder learns what's needed before getting stuck midway through. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Define the rollback procedure as a first-class section written before the forward change, not improvised after something goes wrong. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Add post-condition checks at the end of each phase so the runbook itself tells the operator whether that phase actually succeeded before moving on. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Writing steps as descriptive prose instead of atomic, verb-first, copy-pasteable commands with no placeholders left for the reader to fill in under pressure. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Treating rollback as an afterthought written only after something breaks, instead of a first-class section defined and rehearsed before the change ships. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Skipping the "fresh machine" test, so a runbook that works fine for its author, who has undocumented local state or context, fails for anyone else who follows it literally. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Omitting post-condition checks, so a responder can't tell whether a step actually succeeded before moving to the next one, letting a failure compound silently. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Known issues
- A runbook decays the moment the underlying system changes — a config default, a service name, an API — unless it has an assigned owner and a review cadence forcing periodic revalidation. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Decision points with vague judgment calls instead of measurable thresholds produce inconsistent responses between different on-call responders handling the same situation. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Even a well-written runbook can't fully substitute for understanding the system — it optimizes for correct execution under pressure, not for diagnosing genuinely novel failures the runbook didn't anticipate. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Copy-pasteable commands with hardcoded values, rather than clearly marked parameters, can be dangerous if reused in the wrong environment, such as running a production command against staging, without a responder double-checking context first. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- The fresh-machine test (frontier)
- Atomic numbered steps with verb-first imperatives (frontier)
- You-are-here markers and progress anchoring (frontier)
- Prerequisites block at the top (frontier)
- Rollback as a first-class section (frontier)
- Decision points with measurable thresholds (frontier)
- Post-condition checks (assertions) (frontier)
- Ownership, review cadence, and metadata (frontier)
- Copy-pasteable commands and no placeholders in prose (frontier)
- Common runbook anti-patterns (frontier)
Frontier under this node: Atomic numbered steps with verb-first imperatives, Common runbook anti-patterns, Copy-pasteable commands and no placeholders in prose, Decision points with measurable thresholds, Ownership, review cadence, and metadata, Post-condition checks (assertions), Prerequisites block at the top, Rollback as a first-class section, The fresh-machine test, You-are-here markers and progress anchoring