How to write better surveys: 6 questions to ask about your answer categories before launch

Better surveys are surveys where the answer categories — not just the questions — have been deliberately designed to capture what respondents actually think. Most survey errors hide in the answer options: missing middle ground, unbalanced positive and negative options, scales that change halfway through, response choices that don't cover every reasonable answer. The questions themselves can be perfectly worded and the data still ends up unreliable.

This article gives you six concrete checks to run on your answer categories before you launch — plus four specific words to flag in your question wording, where bad surveys most often go wrong. As Winston Churchill and Benjamin Franklin both put it, in their own ways: "To fail to plan is to plan to fail." Nowhere is that more true than in survey design, where a one-hour planning investment can return value many times over once the data starts arriving.

Answer category design is the structured process of choosing the response options for each survey question — including scale length, label wording, neutral and "don't know" options, and the balance between positive and negative choices. Effective answer categories are mutually exclusive (no two options overlap), collectively exhaustive (every reasonable answer has an option), and balanced (positive and negative choices are equally represented). Most survey errors hide in the answer categories rather than in the questions themselves.

The questionnaire goes out. The questions read fine. The platform works. Two weeks later, the data is in — and 60% of respondents picked "Good" on the canteen question, because the only other option was "Bad". The middle of the workforce, the people who think the canteen is fine but not great, picked "Good" because they had nowhere else to go. The result: an artificially high satisfaction score, a leadership team that thinks canteen reform isn't needed, and a measurement programme that quietly produces wrong answers for a year. Bad answer categories don't break the survey. They just make the data lie quietly. 

Answer categories produce more survey errors than questions because they are the constraint respondents have to fit their actual feeling into. A good question gives the respondent room to think; a bad answer category forces them to pick a response that doesn't match what they actually think. The result is data that looks clean but distorts what the survey was supposed to capture — and the distortion is usually invisible in the final numbers.

The six checks below are the discipline that catches the most common answer-category errors before they reach respondents. Run them on every survey before you launch. The hour they take is the cheapest insurance you can buy on the data quality of the next 12 months of measurement.

1. Do the answer categories match the question?

In strictly practical terms: can the respondent actually answer what's being asked? Most of the time the answer is yes — but not always. A classic error is a survey trying to draw conclusions about canteen satisfaction with only two response options: "Good" or "Bad". The middle ground — "In between", "Neither good nor bad" — has been forgotten. That matters: respondents who feel they sit in the middle, with no way to express it, will at best skip the question and at worst pick an answer that isn't actually true. If enough respondents do the same, the entire survey skews.

2. Are the answer categories easy to scan and pick from?

In other words: can the respondent quickly see all the options and pick the one that fits them best? If yes, you're good. If no, it's worth reconsidering the approach. That might mean changing the answer category from a battery of options to a scale, or using symbols — smileys and other icons often make scanning easier.

Solutions vary depending on the question's character, but once you're aware of the issue, the right answer usually presents itself. Surveyxact support and consultants are available if you get stuck or want a second opinion — and there is a deep knowledge base behind the platform for self-service investigation.

3. Are there enough answer options to cover all reasonable responses?

This one is a blend of the first two checks. Easy scanning is critical, but so is making sure the options cover all the reasonable answers respondents might want to give — without breaking scannability.

Take the canteen example again. It isn't only the in-between respondents who get overlooked when there are only "Good" and "Bad" options. What about the respondents who love the canteen, or the ones who can't stand it? Adding "Very good" and "Very bad" gives those respondents a place to sit. The art is finding the balance: enough options to cover every reasonable answer, few enough to stay scannable, all of them leading to reliable results.

4. Are the answer options balanced — not pushing respondents toward one side?

One of the things that most often pulls survey data off-true is answer options that push respondents — too high or too low — toward a particular answer. In a perfect world, every respondent would have a clear, considered position on every question. In practice, opinion strength varies enormously: some respondents have a strong view on the canteen, others have never thought about it.

In that situation, it matters that the question wording and the answer options don't push respondents in either direction. If respondents don't say what they actually feel, the final result is distorted, and the validity and reliability of the survey come into question. Balance is the discipline that keeps the data honest.

5. Are the categories mutually exclusive and collectively exhaustive?

Two technical terms that solve a lot of practical problems. Answer categories are mutually exclusive when there is balance between positive and negative options — for example, if you offer "Very satisfied" and "Satisfied", the negative end of the scale should be equally represented with "Dissatisfied" and "Very dissatisfied". Without that symmetry, the scale itself biases the score.

Answer categories are collectively exhaustive when respondents always have an option that fits their answer. In practice, that usually means including "Don't know" or "Prefer not to answer". The aim is that respondents don't drop out mid-survey — or, worse, answer untruthfully — because the categories don't include their actual position.

Mutually exclusive vs collectively exhaustive — the two-rule check

• Mutually exclusive: the categories don't overlap, and every option is balanced by an equivalent on the other end of the scale.
• Collectively exhaustive: every reasonable answer has a category — including "Don't know" or "Prefer not to answer" where the topic warrants it.

6. Does every question serve a purpose?

Before you put the final full stop on the questionnaire, take time to review the questions one last time. Does each question serve the survey's primary purpose? Have you narrowed the questionnaire to what actually matters? Are you asking the same thing twice in different words?

For each question, ask yourself — and ideally a colleague:

• Is this question necessary?
• Is the question leading?
• Is the question asking more than one thing at the same time?
• Could the question be misunderstood?
• Is the question too simple — or too complex — for the target audience?
• Does the question match the data collection method being used?
• Is the question at the level of measurement your analysis needs?

Bonus: 4 words that signal trouble in your question wording

If you can tick all six checks above, you're well on the way to a successful survey. But there is one last layer to be aware of: four specific words that show up disproportionately often in questions that produce unreliable data. Questions and the way you phrase them are, after all, the glue that holds the survey together.

Be especially careful when these four words appear in your questions:

• And — are you sure you're not asking two questions at once? "Are you satisfied with our service and pricing?" should be split into two questions.
• Or — same situation as "and". "Did you find the documentation useful or up-to-date?" mixes two distinct evaluations.
• If — consider whether the question is becoming too hypothetical. Hypothetical questions produce hypothetical answers, which rarely match real behaviour.
• Not — if you're looking for a yes/no answer, negation makes the question hard to answer cleanly. "Are you not satisfied?" produces "yes I'm not" or "no I'm not" — neither is unambiguous.