Survey Design Part 1: Planning for your survey – A review of Designing Quality Survey Questions (2019) by Robinson and Firth-Leonard
This article is rated as:
You’ve determined that you need a survey to gather information from a specific population you serve. But where do you start? I recently read the book Designing Quality Survey Questions (2019) by Sheila B. Robinson and Kimberly Firth Leonard, which if you are still a survey design novice, would be a great resource.
In the book, Robinson and Leonard discuss the iterative process of designing a survey. The book is divided into three parts: planning and predesign, drafting questions, and finalizing the survey. Each of these parts is seen as a loose phase in survey design and development; I emphasize loose as this is an iterative process. To best cover the three main components of survey design, I have chosen to follow a similar flow. I found the drafting questions and the finalizing of the survey sections more useful in the drafting of a survey, however, the planning and predesigning section is still an important aspect in survey design and does need to be covered.
Why quality survey design?
The book begins with why survey design is important before the authors even discuss how to design quality surveys. They emphasize the need for quality surveys to get people to want to complete your survey as people are now inundated with survey requests. Part of Robinson and Leonard’s aim in this book is to help you improve on your survey design which in turn would help improve respondent experience and data quality. Throughout the book, they emphasize respondent experience over ease of analysis. The authors see respondent experience as imperative to quality survey design. I see their point here, as frequently I haven’t finished a customer experience survey because my choices didn’t make sense, they had too many textboxes, or the survey itself was too long.
Articulate the purpose
In the planning and predesigning phase, the aim is to understand and articulate the purpose of the survey, along with what the survey can measure, and start to understand the survey respondents. At this phase, the evaluator would determine what knowledge they hope to gain through the use of the survey. The authors included evaluation question(s) that the survey would inform, how the information will be used, and who will use that information as part of the clearly articulated purpose of the survey.
Ensure the survey is the right tool
After articulating the purpose, this is when the evaluator would determine if a survey was the right tool to gather the information. Robinson and Leonard articulate the purpose as 1) understanding why you have chosen to use a survey, and 2) outlining what is planned for the results to help ensure a survey is the correct tool. As they view survey design as iterative, you could reverse these steps if that process made more sense to you. Personally, I think these are really happening at the same time.
The authors then review what can be measured by a survey, including respondent’s attributes, behaviours, abilities, and thoughts. The authors also articulate the advantages and limitations to using the survey tool.
I appreciate the authors spending time outlining the advantages and limitations of the survey tool, to help the evaluator determine if this is indeed the tool for the evaluation questions and potential respondents. I wouldn’t spend time developing a survey when interviews would be a better method.
Survey respondents
Robinson and Leonard devote an entire chapter to understanding respondents, which makes sense to me as the whole purpose of the survey is to gather information from this group, so respondents need to be understood as in-depth as possible by the evaluator. The authors start this chapter with the four cognitive tasks respondents must use in answering surveys: comprehension, retrieval, judgement, and response.
They then examine the potential respondents’ willingness and ability to participate in a survey. They ask questions such as:
Do the types of questions and the nature of the questions encourage respondent participation?
Can the potential respondents remember that information, or understand the questions and the language used?
To help with comprehension, the authors do mention the QUAID (Question Understanding Aid) tool that was developed by the Institute for Intelligent Systems at the University of Memphis. This tool can give some feedback on the language used and may be helpful with comprehension. Robinson and Leonard stress that with any tool they suggest, it doesn’t replace field testing or getting feedback from other people in the process of survey design but may help get your tool closer to a final version prior to pre-testing. The authors cover pre-testing the survey later in the book, after drafting survey questions.
The authors then discuss the importance of understanding the context and culture of the potential respondent population. There are several questions the evaluator should consider, such as:
What is the current political, environmental, economic, organizational, and cultural context?
How does the cultural background of the evaluator differ from that of the respondents and how can the evaluator be respectful and responsive to this?
What about the power dynamics; who had the power and privilege or at least who has the perceived power and privilege?
Overall, I found this book a helpful resource in survey design. This book is great for someone new to survey design and could be a great resource for those more experienced as they do give many examples and list other resources for further information at the end of every chapter. I found the later chapters around crafting survey questions and drafting the survey more useful and will review these chapters and any tips and tricks that would be useful in a later article.