What you need to know about member checking
This article is rated as:
Member checking is a technique often used with qualitative methods to help validate findings. It can be used in evaluation to help validate, interpret, and analyze findings from interviews, focus groups, and other forms of qualitative data.
While commonly used in qualitative research, especially as it is included in qualitative research checklists, like COREQ, it’s less commonly used in evaluation and we think that should change!
In this article, we’ll review what member checking is and why, when, and how you should use it.
What is member checking?
In essence, member checking is when participants in qualitative research validate their data, checking for accuracy and validity. Member checking can happen informally during data collection when the researcher or evaluator summarizes and confirms their interpretation of what a participant said during data collection. This can look like an interviewer summarizing a participant’s statement mid-interview, allowing the participant to affirm or correct the statement. My personal favourite lead into this is, “So what I’m hearing you say is…”.
This article will focus instead on formal member checking, whereby the researcher or evaluator reaches back out to participants after data collection to check their findings. This method is attributed to Lincoln and Guba, who argued its purpose is to validate, verify, and assess the trustworthiness of results. I’d argue that it can also be used to build trust with your participants, correct assumptions you have made, and lead to new insights and deeper understandings of the data.
Formal member checking can encompass a range of activities and can occur at different points during data collection and analysis.
Immediately after data collection: participants can review their transcripts and are invited to check the accuracy of the data. Participants can be given the opportunity to add, remove, or clarify their statements.
During data analysis: participants can be invited to review preliminary themes and how their quotes fit into these themes, leaving room for further discussion and (re)interpretation of data. Participants can also be involved in the theming and sense-making process.
After data analysis: participants can review draft reports with their quotes and contributions highlighted for their review. Often at this stage, participants are limited in their ability to change their quotes or provide additional context, but it gives them a clear idea on how their data is presented and used and provides an opportunity for participants to review if the right emphasis was placed on topics and themes.
When should you use member checking?
Now you may be thinking, ‘sounds great! How do I know when it’s a good idea to use member checking?’
Consider using member checking when:
You need to build trust with your participants. giving them the chance to edit or clarify what they said can help participants trust you, the evaluator. This can also be a valuable step if you know you’ll need to return to this same group for future data collection.
Your sample is small, and it might be easy to know who said what. Despite your best efforts to maintain confidentiality, it can be difficult if your sample is small. Giving participants the opportunity to redact statements, especially how they will appear in context, can help to preserve their confidentiality, or ensure that they are aware of what will be shared and how.
When the topic or content is sensitive. Member checking can support participant confidentiality, allowing them to review how you’ve used their information.
When you aren’t familiar with the context. Member checking can aid in interpretation and analysis and provide additional context.
You need enhanced validity. Member checking can add rigour and validity. Consider using it when other validation techniques, such as triangulation, are not possible. Using member checking can provide evidence that your interpretation and analysis are appropriate, accurate, and reflect the content of the discussion.
You are conducting a participatory evaluation. Member checking can be a participatory technique to include participants in data collection, analysis, and reporting. Member checking can support community buy-in for the evaluation and its results.
Avoid using member checking when:
You have a short timeline or small budget. Member checking takes extra time and therefore more money. Make sure you have the time and budget to do it justice.
Your participants are short on time or doing so will add undue burden. Member checking adds an additional burden to participants. For those that are busy or do not have a lot of capacity, member checking is another activity you are adding to their plate.
Participants will struggle with the concept of themes. For those of us in the research and evaluation world, the concept of summarizing information into themes is easy to comprehend. Avoid doing member checking using themed or summarized information with groups who may not understand how and why themes are used.
When your participants may have low (English) literacy levels. For those who are not comfortable or able to read well in English, member checking by providing a written summary or transcript is likely not a viable option. Feel free to get creative, though, if member checking is important for this group. Short videos or audio clips may be a way to get in touch with this audience.
You are not able to receive input or incorporate feedback. Member checking rests on the principle that participants can modify the transcripts or analysis. If you are unable to make changes, member checking is not an appropriate tool for that project.
When you won’t be able to do member checking close to the interview. Sometimes it takes a long time to gather and analyze data. Member checking loses its benefits when it is done a long time after data collection has occurred as participants may not recall the purpose or context as clearly.
When there are power imbalances between the evaluator and participant. In cases where the participant may not feel comfortable contradicting the evaluator or the findings, member checking may not provide the value or validity you are hoping for. In some cases, you may be able to use other people or methods to even out the power imbalance.
If reviewing their input may pose a risk to participants. In some cases, asking participants to review their contributions can be distressing, especially if the data are gathered about a sensitive topic. Additionally, if participants don’t see their contributions reflected in a summary or the themes, it may leave them feeling isolated or unheard.
What you need to consider when member checking
I’ve described some of the what, why, and when, but the real question remains, how do you actually do it? There’s a range of options for how to member check. You need to decide on a few key things: what to send them, how you want them to interact with what you’ve sent, what you will do with their contributions, and how much time they have.
Let’s start with what to send them. You can send participants their data in a range of forms, from their raw data (e.g., their transcripts), to a summary of their contribution, or their quotes in themes or in the context of the report.
Next up, is how you will connect with them. Individual member checking can be done through a 1:1 conversation using a set of interview questions, an email asking them to reply with comments, or a survey asking for feedback on themes. Member checking can also be done in a group setting, especially when the data were gathered in a group (e.g., focus groups). You can host a group focus group or discussion, or even structure your session like a data party.
Note: while member checking can be done in a data party format, not all data parties are considered member checking. Member checking is specifically done with those who provided data. Data parties can include those who were not involved in data collection to support interpretation.
Next, you need to be clear with what you expect them to do with the information. Are they able to ask for information to be removed or to edit a quote? Can they offer additional context? What if they disagree with the theme or title?
Next, you need a plan for what you are going to do with the feedback. This should be determined in advance. Will you make the edits and re-send them for approval? What will you do with conflicting information? What happens if they don’t respond? Who has the final say about the findings and interpretation?
Finally, be clear on timelines. You should give participants a clear indication of when they should expect to receive the information you are asking them to check and how long they will have to provide feedback. The last thing you want is for someone to be away or not prepared to set aside time to review what you are provided.
In the comments, let us know about a time you’ve used member checking. How did it go?