The “mixing” in mixed methods
This article is rated as:
In evaluation, we use multiple types and sources of data, diverse methods of collection, or multiple evaluators to answer evaluation questions. Data integration is a way of merging these data from different sources through mixed methods. Data integration can enhance reliability in evaluation findings (e.g., by increasing the ability of findings to be replicated). It can also help to discover contradictions and inconsistencies that otherwise might not have been revealed between different sources and can clarify the results of an evaluation.
The ability to synthesize large amounts of data to identify important information is an essential skill for evaluators. Depending on the scope of the evaluation, we often collect large amounts and different types of data, and we must triangulate them to get to the main evaluation findings. “Mixed methods” is intentionally using one data source with another, with the purpose of triangulating the results, whereas “multiple methods” is simply using different data collection strategies in the same program, but with no intention to "mix" or integrate them.
To give you a simple analogy, “mixed methods” is like mixing coffee and milk together (e.g., latte), while “multiple methods” is having coffee and milk separately. They are both great but very different beverages.
In this article, we discuss how qualitative and quantitative data can be integrated at the study design level, methods, or analysis level.
Data integration at the design level
At the design level, data can be collected concurrently, or one approach can be used to inform the other.
In exploratory sequential design, we can collect and analyze qualitative data and use the findings to inform upcoming quantitative data collection. A good example would be using interview or focus group results to design survey questions. This exploratory approach improves the survey as it helps to focus the questions on topics that are relevant or important to participants.
Explanatory sequential design uses the findings of quantitative data to plan qualitative data collection. For example, a survey finding can be further explored using interviews to understand what, how, and why. This approach often leads to a much richer discussion as the evaluator already understands the underlying issues and can further explore those specific themes in interviews and/or focus groups.
If we conduct the qualitative and quantitative data collection simultaneously in convergent design, the findings from one approach can still inform and drive change in an interactive approach. For example, using interviews and survey findings in multiple phases such that the data interact to inform subsequent versions and the final result. This approach is resource-intensive and requires many cycles of participation from respondents.
Data integration at the methods level
Data integration at the methods level occurs when the qualitative data collection is linked to quantitative in the data collection or analysis.
Data collection can be linked through the sampling frame (connecting) whereby participants for one method can be recruited/invited to participate in another method (e.g., recruiting focus group participants from survey respondents).
I often use this approach in my evaluation practice where I recruit interview or focus group participants through surveys. It is often difficult to reach program participants thus, using one data collection effort to recruit for other methods reduces the burden on participants and minimizes evaluation cost. I use the app Calendly, which works like a dream to schedule interviews as the link to the app can be inserted at the end of the survey. Calendly will automatically show interested participants potential interview times and lets them schedule time that works for them.
Other ways of integration at the methods level include embedding, which is linking data at multiple points (e.g., the use of the first round of qualitative data to understand and control for potential bias in an initial survey and using a second round of qualitative data to further explore survey results. In this example, there are two rounds of qualitative data collection and a survey between them. The evaluator uses the findings from each data collection effort to inform the next one) or bringing them together for analysis (merging).
Data integration at the Interpretation and Reporting level
Data integration at the Interpretation and Reporting level often occurs in one of the following approaches:
Narrative – describing the qualitative and quantitative findings in a report. The evaluator weaves the qualitative and quantitative findings together on a topic-by-topic basis or presents the findings in different sections.
In my evaluation reports, I often do a combination of weaving and presenting in different sections. In the results section, I present the results of administrative data collection, surveys and interviews separately and bring them all together by general themes/topics in the discussion or key takeaway sections.
Data transformation – one type of data is converted into the other type of data, then the transformed data is integrated with the other data and analyzed simultaneously. An example will be transforming the qualitative data into numeric counts and variables using content analysis to integrate with a quantitative database.
Lastly, we can integrate data using joint displays, which incorporates the qualitative and quantitative data through visual means to present new considerations. The example below presents survey and focus group results side-by-side to provide comprehensive information. If you would like more information on joint displays, check out this article.
Whichever approach you choose, integrating qualitative and quantitative data and triangulating the results often helps generate new insights and reliable evaluation results. Evaluators should consider which evaluations would benefit from mixed methods and carefully choose their data integration approach.
Which data integration approach do you use often? Let us know in the comments.
The data integration approaches listed here are a summary of the article “Achieving Integration in Mixed Methods Designs—Principles and Practices” by Michael D. Fetters, Leslie A. Curry, and John W. Creswell.