Surveys are a staple in medical and health professions education (HPE) research, but how often do they meet basic quality standards?
In our 2018 study published in Academic Medicine, we reviewed dozens of published survey instruments to answer that very question. What we found was troubling: nearly 95% of the surveys we analyzed violated one or more fundamental design principles.
Common Flaws That Undermine Survey Results
Surveys are only as good as the questions they ask. But many researchers unknowingly introduce bias and confusion through poor item design. Our study focused on five common pitfalls found in Likert-type survey items:
- Agreement response options
These increase the risk of “acquiescence bias” (people just agreeing with statements regardless of content). - Multibarreled questions
Asking two things at once (e.g., “How effective is communication up and down the chain of command?”) makes it impossible to know what the response actually means. - Unlabeled response options
When only the endpoints are labeled, respondents interpret the middle options differently, reducing reliability. - Unevenly spaced choices
Poor visual layout distorts the response scale and can nudge people toward certain answers. - Mixing substantive and nonsubstantive options
Including “N/A” or “Don’t know” alongside substantive answer choices can throw off results.
What We Found
We analyzed 37 published surveys from top HPE journals. Here’s what we discovered:
- 95% of the surveys had at least one major design flaw
- Only 35% reported any validity evidence
- Only 22% provided reliability evidence
- More than half used agreement-based response options
- Nearly two-thirds included multibarreled items
Even more surprising? These flaws were found in surveys used for peer-reviewed studies published in some of the most prestigious journals in the field.
Why This Matters
Poor survey design doesn’t just negatively affect your data, it can negatively affect your decisions. It can lead to:
- Misleading research findings
- Flawed program evaluations
- Inaccurate feedback loops for learners, clinicians, or institutions
In other words, bad surveys = bad decisions.
What You Can Do
Whether you’re developing a new instrument or using one from the literature, we recommend:
- Following established item-writing best practices
- Pretesting your survey with real users
- Collecting and reporting validity and reliability evidence in your own context
- Avoiding the five pitfalls outlined above
Importantly—if you’re using or adapting someone else’s published survey, don’t automatically assume that survey is a good one!
📩 Want help auditing or improving your survey instrument? Contact Us.
