The Scenario
A master’s student recently sent me their survey questionnaire for review. As I’ve seen in many such cases, there were several substantial issues that, if uncorrected, would compromise the quality of the data collected—and, in turn, the analysis, results, and conclusions.
Fortunately, the survey had just begun, so there was still time to make corrections. This situation reminded me of a professionally designed survey I was asked to complete for a large company. The questionnaire forced respondents to answer every question but failed to provide a viable response option for one item. This made it impossible for me to continue. I emailed the company to highlight the flaw, but it was a stark reminder of how critical it is to design a robust survey instrument from the outset.
The Issues
In the student’s case, the questionnaire had several common but serious flaws:
- Confusion Between Anonymity and Confidentiality:
The questionnaire assured respondents their answers would be both anonymous and confidential because no personal details were collected. However, anonymity means no identifying information is recorded, while confidentiality refers to protecting the collected data. Conflating the two terms can undermine trust in the study. - Non-Exhaustive Sociodemographic Options:
Some sociodemographic items, such as the highest level of education, lacked comprehensive response options. For example, there was no category for “below matriculation.” Similarly, marital status categories were incomplete. Failing to include exhaustive options risks alienating respondents and producing incomplete data. - Non-Mutually Exclusive Responses:
Questions like “Which supermarket do you shop at?” provided a list of supermarkets but allowed only one response. Respondents who shop at multiple stores were left uncertain. Either allow multiple responses or clarify that the respondent should select their primary store. - Unordered and Unbalanced Response Categories:
One question used the categories “Very poor,” “Good,” “Can’t decide,” and “Poor.” This was problematic on two counts:- The categories were unordered, disrupting the natural flow respondents expect.
- The categories were unbalanced; “Very poor” lacked a counterpart like “Very good,” which could bias responses.
- Issues with Likert-Type Items:
Many items measuring attitudes or perceptions using Likert scales contained common errors:- Double-barreled questions about more than one issue (e.g., “How satisfied are you with the quality and price?”).
- Negative phrasing, biased wording, or assumptions about the respondent.
The Solution
Designing a robust questionnaire requires thoughtful planning and testing:
- Pilot the Questionnaire:
Administer your questionnaire to a small sample of respondents to identify and fix any issues before the full rollout. - Use Clear, Simple Wording:
Ensure all items are straightforward and unambiguous. - Create Exhaustive and Mutually Exclusive Response Options:
- Include all possible categories for single-response questions.
- For multiple-response questions, indicate that respondents can select more than one option.
- Balance and Order Response Categories:
Ensure response options are logically ordered (e.g., “Very poor” to “Very good”) and balanced on both ends to avoid bias. - Avoid Common Pitfalls in Likert Scales:
- Avoid double-barreled items—look for the word “and” as a red flag and split the question into two if necessary.
- Refrain from using biased or judgmental terms.
- Don’t make assumptions about respondents’ experiences or perspectives.
The Key Takeaway
A well-designed questionnaire is the backbone of a reliable survey. It ensures the data collected is accurate, comprehensive, and meaningful. Avoid pitfalls by piloting your questionnaire, using precise wording, and ensuring response categories are exhaustive, mutually exclusive, balanced, and ordered. Remember, a good questionnaire respects respondents’ time and provides a solid foundation for your research.