Survey Question Writing
Parent: Writing and Documentation · Published reference · snapshot 2026-09-08 · skill career-and-formal-writing/references/survey-question-writing.md
↓ Facts as markdownall context files
A survey question is a measurement instrument. Bad wording does not just irritate respondents — it injects measurement error that downstream statistics cannot fix.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Overview
- A survey question is a measurement instrument. Bad wording does not just irritate respondents - it injects measurement error that downstream statistics cannot fix. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#overview]
- Default mental model: every question is a hypothesis about what the respondent will read. If two thoughtful readers could parse the stem differently, the question is broken. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#overview]
1. Question-stem hygiene (Dillman)
- Stems must be direct, concrete, mutually exclusive, and answerable in one read: [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#1-question-stem-hygiene-dillman]
- One concept per question - no "and/or" coordination across distinct constructs. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#1-question-stem-hygiene-dillman]
- Common vocabulary - no jargon. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#1-question-stem-hygiene-dillman]
- Concrete time window - "in the last 30 days" not "recently". [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#1-question-stem-hygiene-dillman]
- Specified reference - "your most recent purchase" not "purchases". [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#1-question-stem-hygiene-dillman]
- Symmetric framing - avoid "do you agree that X is good" (loaded). [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#1-question-stem-hygiene-dillman]
2. Likert scales: 5-point vs 7-point
- Default: 5-point for operational customer/employee surveys. 7-point when you need discrimination for regression/factor analysis. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#2-likert-scales-5-point-vs-7-point]
3. Balanced anchors and label coverage
- A balanced scale has the same number of positive and negative points around a neutral midpoint, with labels on every point. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#3-balanced-anchors-and-label-coverage]
- Good: Strongly disagree / Disagree / Neither / Agree / Strongly agree [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#3-balanced-anchors-and-label-coverage]
- Bad: Hate it / Dislike / Neutral / Like / Love (asymmetric intensity) [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#3-balanced-anchors-and-label-coverage]
4. NPS, CSAT, and CES — pick one per question
- Do not modify the NPS stem if you want to compare to industry benchmarks. NPS scoring: 0–6 = Detractors, 7–8 = Passives, 9–10 = Promoters. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#4-nps-csat-and-ces-pick-one-per-question]
5. The agree-disagree anti-pattern (Saris & Gallhofer 2014)
- "Do you agree or disagree that [statement]?" invites acquiescence bias (respondents lean toward "agree"). [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#5-the-agree-disagree-anti-pattern-saris-gallhofer-2014]
- Anti-pattern: "Do you agree that the website is easy to use?" (5-pt agree/disagree) [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#5-the-agree-disagree-anti-pattern-saris-gallhofer-2014]
- Better: "How easy or difficult is the website to use?" (Very difficult ... Very easy) [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#5-the-agree-disagree-anti-pattern-saris-gallhofer-2014]
- Item-specific scales reduce systematic error and increase reliability. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#5-the-agree-disagree-anti-pattern-saris-gallhofer-2014]
6. Double-barreled and leading questions
- Double-barreled = one question, two concepts. Split it. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#6-double-barreled-and-leading-questions]
- Bad: "How satisfied are you with the price and quality of the product?" [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#6-double-barreled-and-leading-questions]
- Fix: Two questions, one for price, one for quality. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#6-double-barreled-and-leading-questions]
- Leading = stem prejudges the answer. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#6-double-barreled-and-leading-questions]
- Bad: "How helpful was our amazing support team?" [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#6-double-barreled-and-leading-questions]
- Fix: "How would you rate the support you received?" (Very poor ... Very good) [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#6-double-barreled-and-leading-questions]
7. "Don't Know" vs "Neutral" vs forced-choice
- Neutral midpoint = respondent has an opinion but it is centered. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#7-dont-know-vs-neutral-vs-forced-choice]
- "Don't Know" / "Not applicable" = respondent lacks the information. Place it visually offset from the scale (Dillman) so satisficers don't select it by default. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#7-dont-know-vs-neutral-vs-forced-choice]
- Forced-choice = use only when you genuinely need a side. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#7-dont-know-vs-neutral-vs-forced-choice]
8. Ordering effects
- Question order: early questions can prime later ones. Put sensitive/demographic questions last. Put the headline metric (NPS/CSAT) before drill-downs. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#8-ordering-effects]
- Response option order: randomize unordered option lists across respondents. Never randomize a Likert. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#8-ordering-effects]
- Matrix straight-lining: long batteries of similar Likert items invite straight-lining. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#8-ordering-effects]
9. Open- vs closed-ended
- Closed-ended = the default for quantitative analysis, benchmarking, large samples. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#9-open--vs-closed-ended]
- Open-ended = use sparingly (1–2 per survey). Place after the closed question that primed the topic. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#9-open--vs-closed-ended]
10. Mobile constraints
- Stem ≤ 20 words; aim for ≤ 12. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#10-mobile-constraints]
- Scale ≤ 7 points (5 preferred). [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#10-mobile-constraints]
- Avoid grids/matrices on mobile - collapse to single-question-per-screen. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#10-mobile-constraints]
- Total survey ≤ 5 minutes for transactional touchpoints. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#10-mobile-constraints]
NPS (canonical, do not modify the stem)
- > How likely is it that you would recommend [Company/Product] to a friend or colleague? > 0 (Not at all likely) - 10 (Extremely likely) [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#nps-canonical-do-not-modify-the-stem]
Generic attribute rating (item-specific, preferred over agree-disagree)
- > How would you rate the [speed / clarity / accuracy] of the response you received? > Very poor - Poor - Fair - Good - Excellent [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#generic-attribute-rating-item-specific-preferred-over-agree-disagree]
Anti-Patterns
- Agree-disagree everything. Causes acquiescence bias. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
- Double-barreled stems. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
- Loaded adjectives in stems. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
- Endpoint-only labels on 7-pt scales. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
- Random Likert order. Never randomize ordered response options. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
- Modifying the NPS stem. Breaks benchmark comparability. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
- Stacking NPS + CSAT + CES + 10 drill-downs. Pick a primary metric per touchpoint. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#anti-patterns]
References
- Dillman, D. A. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th ed.). Wiley. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#references]
- Saris, W. E., & Gallhofer, I. N. Design, Evaluation, and Analysis of Questionnaires for Survey Research (2nd ed.). Wiley. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#references]
- Reichheld, F. F. "The One Number You Need to Grow." Harvard Business Review, 2003. [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#references]
- Pew Research Center. "Writing Survey Questions." https://www.pewresearch.org/writing-survey-questions/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/survey-question-writing/#references]
Where this helps
- Writing a Likert-scale question and deciding between 5-point (default for operational customer/employee surveys) and 7-point (better discrimination for regression/factor analysis). [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Choosing among NPS, CSAT, and CES for a single question, where mixing metrics or modifying the canonical NPS stem breaks comparability to industry benchmarks. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Catching a double-barreled question ("how satisfied are you with the price and quality...") before it goes out and silently injects measurement error. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Deciding where to place "Don't Know" versus a neutral midpoint versus forcing a choice, especially for respondents who genuinely lack the information to answer. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
How to apply this
- Apply Dillman's stem hygiene before shipping any question: one concept per question, common vocabulary, direct and concrete wording that two thoughtful readers can't parse differently. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Prefer item-specific rating stems ("How easy or difficult is the website to use?") over the agree-disagree format ("Do you agree that..."), since agree-disagree wording invites acquiescence bias. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Order questions deliberately: put the headline metric (NPS/CSAT) before drill-downs, and put sensitive or demographic questions last so they don't prime earlier answers. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Design for mobile constraints explicitly — keep stems to 20 words or fewer, scales to 7 points or fewer, and avoid grid/matrix layouts that don't collapse well to a single-question-per-screen format. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Writing agree-disagree stems by default, which invites acquiescence bias — respondents lean toward "agree" regardless of the underlying attitude. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Double-barreled questions that bundle two distinct concepts (price and quality) into one item, making the response impossible to interpret cleanly. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Randomizing a Likert scale's response order, when Likert options should stay in their natural order — only unordered option lists should be randomized across respondents. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Modifying the canonical NPS stem, which breaks comparability to industry benchmarks that assume the standard wording and 0-10 scale. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Limitations
- Even a well-designed closed-ended question aggregates well but loses nuance a genuinely important open-ended answer could have captured — open-ended items should stay sparse (1-2 per survey) rather than being abandoned entirely. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Long batteries of similar Likert items invite straight-lining (respondents picking the same point down a whole matrix), which no amount of stem wording alone fully prevents. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Mobile constraints (short stems, small scales, no grids) can force real content or nuance out of a question that would be fine on desktop, creating a tradeoff between response quality and completion rate. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- "Don't Know" and a true neutral midpoint measure different things — lack of information vs. a centered opinion — and conflating them in the same response option corrupts the resulting data regardless of how well the stem itself is worded. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Where this helps
- Writing a Likert-scale question and deciding between 5-point (default for operational customer/employee surveys) and 7-point (better discrimination for regression/factor analysis). [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Choosing among NPS, CSAT, and CES for a single question, where mixing metrics or modifying the canonical NPS stem breaks comparability to industry benchmarks. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Catching a double-barreled question ("how satisfied are you with the price and quality...") before it goes out and silently injects measurement error. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Deciding where to place "Don't Know" versus a neutral midpoint versus forcing a choice, especially for respondents who genuinely lack the information to answer. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
How to apply this
- Apply Dillman's stem hygiene before shipping any question: one concept per question, common vocabulary, direct and concrete wording that two thoughtful readers can't parse differently. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Prefer item-specific rating stems ("How easy or difficult is the website to use?") over the agree-disagree format ("Do you agree that..."), since agree-disagree wording invites acquiescence bias. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Order questions deliberately: put the headline metric (NPS/CSAT) before drill-downs, and put sensitive or demographic questions last so they don't prime earlier answers. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Design for mobile constraints explicitly — keep stems to 20 words or fewer, scales to 7 points or fewer, and avoid grid/matrix layouts that don't collapse well to a single-question-per-screen format. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Writing agree-disagree stems by default, which invites acquiescence bias — respondents lean toward "agree" regardless of the underlying attitude. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Double-barreled questions that bundle two distinct concepts (price and quality) into one item, making the response impossible to interpret cleanly. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Randomizing a Likert scale's response order, when Likert options should stay in their natural order — only unordered option lists should be randomized across respondents. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Modifying the canonical NPS stem, which breaks comparability to industry benchmarks that assume the standard wording and 0-10 scale. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Limitations
- Even a well-designed closed-ended question aggregates well but loses nuance a genuinely important open-ended answer could have captured — open-ended items should stay sparse (1-2 per survey) rather than being abandoned entirely. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Long batteries of similar Likert items invite straight-lining (respondents picking the same point down a whole matrix), which no amount of stem wording alone fully prevents. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Mobile constraints (short stems, small scales, no grids) can force real content or nuance out of a question that would be fine on desktop, creating a tradeoff between response quality and completion rate. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- "Don't Know" and a true neutral midpoint measure different things — lack of information vs. a centered opinion — and conflating them in the same response option corrupts the resulting data regardless of how well the stem itself is worded. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- Likert scale design (frontier)
- NPS / CSAT / CES choice (frontier)
- Question-stem hygiene (Dillman) (frontier)
- Agree-disagree anti-pattern (Saris & Gallhofer) (frontier)
- Double-barreled and leading question detection (frontier)
- Anchor labels and ordering effects (frontier)
- Mobile survey constraints (frontier)
Frontier under this node: Agree-disagree anti-pattern (Saris & Gallhofer), Anchor labels and ordering effects, Double-barreled and leading question detection, Likert scale design, Mobile survey constraints, NPS / CSAT / CES choice, Question-stem hygiene (Dillman)