Incentives for participation in evaluation
This article is rated as:
I’ve been thinking recently about bias in evaluation methodology. There’s so many different types of bias it’s hard to keep them all straight, never mind mitigating them all!
One particular area has me interested:
What bias does offering an incentive to participate introduce?
So, let’s broaden the question and explore incentives for participation generally (don’t worry, we’ll cover bias too).
As evaluators, one of our primary activities is data collection. Some of the most common methods include asking other people to share their thoughts or experiences in surveys, interviews, or focus groups. Sometimes it can be difficult to get people to agree to participate. It’s very enticing to offer an incentive to boost that response rate or sample size.
Why offer an incentive?
Many argue, myself included, that those with lived experience are experts and knowledge keepers, and should be compensated for sharing their expertise. So, incentives can serve a moral purpose, offering compensation in exchange for expertise.
Incentives can also serve the function of attempting to mitigate non-response bias. Theoretically, by increasing response rates, your data will be that much more robust and representative of the population you are hoping to report on.
Is cash king?
The two most common forms of incentives are cash or gift cards. So, which should you use, when, and why?
When I was a graduate student, I ran a study that collected data from individuals experiencing challenges with substance use. The Research Ethics Board (REB) at the university would not let me offer cash as an incentive to participate. It never sat well with me. Ethically, who am I, as a researcher or evaluator, to decide whether a participant “deserves” cash or not? Who am I to declare that they can only spend earnings at gift card locations? Besides, there’s nothing stopping anyone from re-selling gift cards for cash. It seemed like a misstep to me.
More recently I offered a cash incentive to participants for an interview. Many of those participants voluntarily expressed appreciation for the cash support. That project felt better for me.
So, morally, I am a believer that cash is appropriate. Functionally, research says that money is more motivating than other gifts. So, yes, cash may actually be king.
Low budget options
What happens if you don’t have the budget for cash or gift cards? There are still ways to offer incentives. Alternatives could be:
Access to products or services (e.g., 1 month of free access to the services of the organization)
Company swag or merchandise
Discounts or coupons (e.g., a 50% discount on your next program registration)
Or perhaps you don’t have the budget to offer everyone an incentive, but you can afford something. In these cases, I’ve used the “chance to win” approach. This is more common in surveys, whereby filling out a survey the participant is entered into a draw to win something, usually a gift; but there’s no reason this couldn’t also be cash.
CAUTION: Make sure that your cash-alternative prizes (e.g., gift cards or products) don’t introduce new forms of bias! If you advertise that participants get gift cards to a specific store or restaurant you may be biased toward people who frequent those businesses. If you advertise that participants get a discount on products or services, you may be biased toward people interested in those products or services.
What about coercion?
I said we’d talk about bias. When offering an incentive, the risk is that it introduces coercion: the practice of persuading someone to do something. It’s a fine balance between motivating someone to spend their time sharing their experience with you in exchange for a token of appreciation, versus inducing a feeling of being compelled because the incentive is so enticing. Coercion exists when a participant accepts otherwise intolerable risk to receive the incentive. I think this risk is real, however, some research has shown that any risk associated with participation is not influenced by incentives. This study found no relationship between the size of the risk and the size of the incentive.
It may be impossible to prove that an incentive isn’t coercive. The best way to protect participants from risk isn’t by making changes to your incentive, but by examining policy, protocol, and consent within your project or program. Probably the best mitigation strategy is informed consent, letting participants know about their rights to refuse, clearly outlining any risk to participation, and describing how participation does or does not affect their access to services. Related to this, refusing access or service based on participation is definitely coercive!
What impact does offering an incentive have on your data?
One reason to offer an incentive is to increase the response rate. Increasing response rates should improve your data quality through more valid, representative data. However, there are negative impacts as well – including introducing profile bias based on the specific incentive (as cautioned above). Incentives could also change the responses themselves, either because participants feel compelled to respond positively to receive the incentive, or with even more subtle changes, like improved mood. However, some research suggests that these influences are unlikely.
Ideally, we want people to participate because they care about the outcome, or they have a vested interest in the program or organization. However, motivations to participate cover 3 primary reasons: 1) altruism 2) topical reasons, e.g., the desire to share a specific positive or negative experience, or interest in the topic or 3) egoist reasons, i.e., for the incentive. Most evaluations probably have a mix of these motivations. It can be difficult or impossible, and probably unnecessary to tease them apart. Ensuring strength in your data collection tools and processes will help to mitigate any risk associated with egoist participation.
What is participation worth?
A colleague of mine recently had to counsel a client on whether $100 for survey participation was coercive or not. While it’s not common to have to debate if the incentive is too much, it’s still a good lesson in ethical practice. And begs the question, what is participation worth?
There’s no one answer. In fact, there isn’t even a “right” answer. It depends on several factors including:
time burden: is it a 5-minute survey, or a 3-hour focus group?
how intense is the participation process?
how specific or niche, and therefore possibly difficult to reach, is your target participant group?
how specialized is the knowledge or experience the participant group brings?
your project budget!
The higher the burden or intensity, or the more specialized the nature of participation, the higher the perceived value.
Wellesley Institute, out of Toronto, did some exploration in 2018 and found that in research (i.e., not evaluation), participants were paid, on average, $30 for an interview, $25 for a focus group, and $20 for a survey. In my own experience, this is pretty close in evaluation as well. Depending on budget and level of burden (i.e., time commitment) I’ve offered up to $50 for interviews. Wellesley suggests that a good rule of thumb is $25 per hour of participation.
Appropriateness of incentives
While cash may be king, there are other considerations when deciding on incentives. I’ve recently been working on a project where paying for services may introduce trauma for some participants. I’ve worked closely with this project team to develop a plan where the participants are compensated for sharing their experiences, but without the transactional nature of giving an incentive after participation. Specifically, persons with lived experience are invited to participate in an interview; regardless of participation, they are also offered extra support to meet any emergent needs they may have. That extra support is not based on their participation, but on the invitation.
Vulnerable populations may respond to incentives differently, by being more likely to accept risk. Vulnerable populations are also frequent target populations for research and evaluations. This is a great resource that offers more guidance on compensation for persons with lived experience, including: discussing compensation clearly and upfront, offering options, paying in cash, and offering to pay for additional costs (like child care).
Depending on your participant population there are groups and cultures where exchanging gifts is an act of respect. Consider reviewing your incentive practice from a lens of diversity and intersectionality.
Finally, there is some grey area about who is offered an incentive. For example, you are evaluating a program, commissioned by the organization that runs the program. Part of your evaluation plan is to interview the program participants. I think we can safely agree that offering some form of incentive is justified. Part of the same evaluation plan is to interview the organization leaders and the facilitators of the program. Does it make sense to offer them an incentive? Or is it “part of their job” to participate? To pay them for participation is essentially paying them out of their own budget. In my experience, I have differentiated between those participating in a professional capacity (i.e., not incentivizing staff) and those volunteering to participate (i.e., incentivizing lived experiences or program participants).
A key to ensuring your incentives are appropriate is an open dialogue with your project team, which hopefully includes a representative from your target population. Engage in your own reflective practice as an evaluator and seek input from your colleagues to identify risks you may have missed.
Other Considerations
To offer an incentive, you may need to collect some contact information. With so much data collection being done virtually these days, many incentives are offered electronically, where email addresses are required. This has at least two impacts:
Is your participant population likely to have an email address and the capacity to retrieve and access an e-transfer or electronic gift card?
Does collecting contact information introduce any ethical risk around identification or breach of confidentiality?
In my experience where most data collection is happening virtually, the idea of bringing food and/or beverages is a moot point, but in the past, I have offered focus group participants snacks and drinks. These “perks” to participating are also an incentive and should be considered thoughtfully.
Arguably outside the discussion of incentives specifically, is reimbursement. Some participants may need to travel or enlist child or elder care to accommodate their participation. It is common practice to offer reimbursement for such expenses, which can take many forms, including cash, bus tickets, cab fare, mileage payment, etc.
I’ve had discussions that the incentive isn’t an incentive, per se, but a “thank-you”. This can be true if the thank-you is given after participation without prior knowledge, or perhaps if the value is so low that it would in no way incentivize anyone. Similarly, some participants have regular hourly rates and can be offered an honorarium to compensate them directly for their time. These distinctions between a thank-you, an honorarium, and an incentive are very blurred. As always, data collection processes and informed consent are the best ways to mitigate risks!