Survey Question Writing

Parent: Writing and Documentation · researched 2026-05-29T19:09:20.118Z· 5 sources · 7 concepts · skill survey-question-writing

A survey question is a measurement instrument. Bad wording does not just irritate respondents — it injects measurement error that downstream statistics cannot fix.

Overview

A survey question is a measurement instrument. Bad wording does not just irritate respondents - it injects measurement error that downstream statistics cannot fix. [source]
Default mental model: every question is a hypothesis about what the respondent will read. If two thoughtful readers could parse the stem differently, the question is broken. [source]

1. Question-stem hygiene (Dillman)

Stems must be direct, concrete, mutually exclusive, and answerable in one read: [source]
One concept per question - no "and/or" coordination across distinct constructs. [source]
Common vocabulary - no jargon. [source]
Concrete time window - "in the last 30 days" not "recently". [source]
Specified reference - "your most recent purchase" not "purchases". [source]
Symmetric framing - avoid "do you agree that X is good" (loaded). [source]

2. Likert scales: 5-point vs 7-point

Default: 5-point for operational customer/employee surveys. 7-point when you need discrimination for regression/factor analysis. [source]

3. Balanced anchors and label coverage

A balanced scale has the same number of positive and negative points around a neutral midpoint, with labels on every point. [source]
Good: Strongly disagree / Disagree / Neither / Agree / Strongly agree [source]
Bad: Hate it / Dislike / Neutral / Like / Love (asymmetric intensity) [source]

4. NPS, CSAT, and CES — pick one per question

Do not modify the NPS stem if you want to compare to industry benchmarks. NPS scoring: 0–6 = Detractors, 7–8 = Passives, 9–10 = Promoters. [source]

5. The agree-disagree anti-pattern (Saris & Gallhofer 2014)

"Do you agree or disagree that [statement]?" invites acquiescence bias (respondents lean toward "agree"). [source]
Anti-pattern: "Do you agree that the website is easy to use?" (5-pt agree/disagree) [source]
Better: "How easy or difficult is the website to use?" (Very difficult ... Very easy) [source]
Item-specific scales reduce systematic error and increase reliability. [source]

6. Double-barreled and leading questions

Double-barreled = one question, two concepts. Split it. [source]
Bad: "How satisfied are you with the price and quality of the product?" [source]
Fix: Two questions, one for price, one for quality. [source]
Leading = stem prejudges the answer. [source]
Bad: "How helpful was our amazing support team?" [source]
Fix: "How would you rate the support you received?" (Very poor ... Very good) [source]

7. "Don't Know" vs "Neutral" vs forced-choice

Neutral midpoint = respondent has an opinion but it is centered. [source]
"Don't Know" / "Not applicable" = respondent lacks the information. Place it visually offset from the scale (Dillman) so satisficers don't select it by default. [source]
Forced-choice = use only when you genuinely need a side. [source]

8. Ordering effects

Question order: early questions can prime later ones. Put sensitive/demographic questions last. Put the headline metric (NPS/CSAT) before drill-downs. [source]
Response option order: randomize unordered option lists across respondents. Never randomize a Likert. [source]
Matrix straight-lining: long batteries of similar Likert items invite straight-lining. [source]

9. Open- vs closed-ended

Closed-ended = the default for quantitative analysis, benchmarking, large samples. [source]
Open-ended = use sparingly (1–2 per survey). Place after the closed question that primed the topic. [source]

10. Mobile constraints

Stem ≤ 20 words; aim for ≤ 12. [source]
Scale ≤ 7 points (5 preferred). [source]
Avoid grids/matrices on mobile - collapse to single-question-per-screen. [source]
Total survey ≤ 5 minutes for transactional touchpoints. [source]

NPS (canonical, do not modify the stem)

> How likely is it that you would recommend [Company/Product] to a friend or colleague? [source]
> 0 (Not at all likely) - 10 (Extremely likely) [source]

Generic attribute rating (item-specific, preferred over agree-disagree)

> How would you rate the [speed / clarity / accuracy] of the response you received? [source]
> Very poor - Poor - Fair - Good - Excellent [source]

Anti-Patterns

Agree-disagree everything. Causes acquiescence bias. [source]
Double-barreled stems. [source]
Loaded adjectives in stems. [source]
Endpoint-only labels on 7-pt scales. [source]
Random Likert order. Never randomize ordered response options. [source]
Modifying the NPS stem. Breaks benchmark comparability. [source]
Stacking NPS + CSAT + CES + 10 drill-downs. Pick a primary metric per touchpoint. [source]

References

Dillman, D. A. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th ed.). Wiley. [source]
Saris, W. E., & Gallhofer, I. N. Design, Evaluation, and Analysis of Questionnaires for Survey Research (2nd ed.). Wiley. [source]
Reichheld, F. F. "The One Number You Need to Grow." Harvard Business Review, 2003. [source]
Pew Research Center. "Writing Survey Questions." https://www.pewresearch.org/writing-survey-questions/ [source]

Children

Likert scale design (frontier)
NPS / CSAT / CES choice (frontier)
Question-stem hygiene (Dillman) (frontier)
Agree-disagree anti-pattern (Saris & Gallhofer) (frontier)
Double-barreled and leading question detection (frontier)
Anchor labels and ordering effects (frontier)
Mobile survey constraints (frontier)

Frontier under this node: Agree-disagree anti-pattern (Saris & Gallhofer), Anchor labels and ordering effects, Double-barreled and leading question detection, Likert scale design, Mobile survey constraints, NPS / CSAT / CES choice, Question-stem hygiene (Dillman)

← the whole tree · 3D view· how to read this page