Evaluation in a Low-resource Setting: Strategies for Success
This article is rated as:
“Start where you are. Use what you have. Do what you can.”
Working in the evaluation field is appealing in that it can take place across various sectors, systems and geographic locations. One area of particular interest for some evaluators, like myself, is program evaluation in low-resource settings (LRS). LRS, sometimes referred to as “resource poor” or “resource strained,” indicates countries or regions that lack the financial means to cover the costs associated with infrastructure, healthcare and/or trained professionals – as well as other system or societal needs. These evaluations are often requested by funders or granting agencies as a means of providing evidence for effective use of funds. Likewise, evaluations are particularly important in LRS as programs need to operate as efficiently as possible due to limited human and material resources.
Program evaluations in a LRS can be challenging in that the program staff may not have the skills or capacity to see it through. However, agencies often require formal evaluation of their funded programs which may lead to the hiring of a contract evaluator. Take USAID for example, they provide funding to LRS programs all over the world and emphasize the need to embed evaluation in program planning. USAID values evaluation so much that they have developed policies to enforce and support agencies in meeting evaluation requirements (see USAID Evaluation Policies).
In my (very biased) opinion, I think that LRS evaluations can be complex, but in the long run will help programs as well as the population and country in which they take place. Moreover, evaluation is often viewed as something that must be done for funders, but in many/most cases is just as valuable to the program itself and it provides a source of evidence to inform program planning. As evaluators we need to advocate for the use of evaluations for both funders and their funded programs – the best way is to show what we can do by conducting meaningful evaluations. We already have the skillset to conduct evaluations of all shapes and sizes (impact, development, process, etc.) – but in a LRS we have to be a bit more creative and thrifty in our approach.
As such, I offer four strategies for success when evaluating within a LRS: Be Tech Savvy, Consider Capacity, Be Ready to Adjust, and Use Yourself as a Resource. As you explore these strategies, I am confident that many will already be a part of your practice in all settings (not just LRS). However, I would encourage you to lean in to the strategies even more when you find yourself in a LRS. Lastly, all evaluations in both high- and low-resource regions are susceptible to being thrown curveballs (take COVID-19 as an example) – meaning that as evaluators we need to be ready to engage new strategies on short notice.
Be Tech Savvy
So you find yourself in a LRS or perhaps a very low budget evaluation, but you still need to exchange information and data with stakeholders. In this case, you could even sub out LRS for “pandemic” and the following advice will still apply.
Even if the program has been using the same spreadsheet since 1995 or has a VPN that takes what seems like 2 days to connect - try to utilize existing systems. When capacity is limited, implementing new systems may result in lost time or frustration.
If you find yourself in a setting where staff are eager to have better software or have requested new systems for their information – start by being proud of your forward-thinking team/client but be cautious of available capacity for both finances and time. What I mean by this is to utilize free (or low cost) and easily accessible software before signing the organization up for a pricey software package.
Sharing Information: Consider free tools such as G Suite or DropBox to share files and work on documents (reports, spreadsheets, presentations) simultaneously. These are less expensive (or free for basic accounts) compared to platforms such as Office 365. Note that many of these tools have offline capability meaning that you can work on the documents even when internet connection is limited.
Communicating: The COVID-19 pandemic showed us the many available resources for connecting. In my experience, many of the free services are just as good. Zoom or GoToMeeting are popular, but only offer a free trial period or require a subscription for longer meetings. Alternatively, well-known platforms such as Skype, Google Hangouts (part of G Suite), or even Facebook through their Messenger Rooms can be used for free video calls. WhatsApp and Slack are also great apps for communicating with teams or clients. These are just a few examples of the many communication tools available to evaluators.
Consider Capacity
Evaluations can often be perceived as a capacity strain, whether it be through using staff time or the cost of hiring an external consultant. Given the capacity limitations inherent to LRS, evaluators should keep capacity at the forefront of their minds from the first meeting to the final iteration of the report. Some examples of capacity considerations during an evaluation include:
Developing the work plan and timeline: Try to plan meetings so that they occur only when necessary and include relevant stakeholders. Regular check-in meetings may not be helpful and may take staff away from program delivery - consider 1-2 page status reports instead. Calendly is a great tool to help with booking these meetings.
Creating evaluation plan/questions: We often evaluate efficiency of programs, but it would also be helpful if evaluators put more of an emphasis on capacity or even a sub-section focused on capacity. Develop questions that are feasible in a LRS and likely to uncover actionable findings (not just funder-mandated metrics). Check out Eval Academy’s How to Write Good Evaluation Questions for guidance on writing evaluation questions.
Collecting data: Try to pull from existing data, add questions to regularly administered surveys, or plan focus groups for days where stakeholders are already in the same location. If appropriate, consider methodologies such as Participatory Action Research (PAR) to build evaluation/research capacity of staff (Baum et al., 2006).
Presenting findings & recommendations: Include capacity building or capacity considerations for all recommendations – after all, recommendations are not likely to be adopted or sustained without capacity. Here’s some ideas about how to write those recommendations.
Whether you are an internal or external evaluator in a LRS, the program or agency should feel like you contributed capacity whether it be through the new knowledge, sound recommendations or development of internal evaluation capacity.
Be Ready to Adjust
As evaluators we like plans and go into projects with a clear timeline and guide of what we would like to accomplish. Does it always go as planned? Definitely not! In all engagements we need to be ready to pivot or adjust our plan – this is even more true for evaluations in LRSs.
As I write this article, we are still deep in the uncertainty of COVID-19. Although many evaluators are already comfortable with remote working, many aspects of conducting an evaluation have had to change. Couple this with the complexities of a LRS and there is no choice but to be flexible and ready to adjust evaluations accordingly.
I expect that evaluators will have many successes (and some failures) to share about their evaluation practice during these times. For now, I will offer a couple suggestions that I have found to be helpful working on a LRS project during a pandemic.
Information gathering: Focus groups, interviews and surveys may need to be facilitated online. Likewise, program staff may need to extract data and share with the evaluator remotely. Workshop facilitation may require the evaluator to search for facilitation tools embedded in video chat platforms (such as surveys) or collaborative tools (such as a shared whiteboard). Here are some tips for Hosting Great Virtual Sessions.
Focus on the most important aspects: Some key evaluation topics or focus areas may need to change. Consider resource constraints or competing priorities (pandemic related or not) and the impact on the program you are evaluating. Ask questions like:
Are the evaluation topics/questions still valid?
Are the various evaluation phases feasible remotely and/or with less access to staff and the population being served?
How can evaluation evidence support the program as it adapts for COVID-19 or other resource constraints?
Revisit the timeline: Expect the timeline to change – whether it is rescheduling some of the meetings, changing them to virtual meetings, or completely revising the project work plan. For example, if the evaluation capacity is severely limited due to a pandemic or other LRS obstacles, look at postponing until there is more capacity. It may be better to postpone or extend the timeline rather than sacrificing the quality of the evaluation.
Use Yourself as a Resource
In a LRS, answers to your questions or material to inform both the planning and execution of an evaluation may not be easily accessible (or in some cases may not be known). As an evaluator for a LRS program, internal or external, you will likely need to be your own resource when it comes to finding data, identifying stakeholders or to develop a general understanding of the program. Hopefully the program or agency at the center of the evaluation will be willing to share all relevant documents and offer some context – but in most cases you will need to be ready to dive in to uncover more. Here are a couple of tips to further explain this strategy:
Hands-on learning: Rather than taking the capacity (resources or staff) to learn about the program or agency being evaluated, consider shadowing or observing the program (activities and meetings). This will prevent the evaluation from eating up too much staff time and I would argue that it will also provide you with a richer understanding of program being evaluated.
Self-led professional development: There may be minimal options for professional development in LRS, especially if you are working as an internal evaluator. Connecting with other evaluators or professionals working on LRS projects is a great place to start. There may be existing Communities of Practice (in-person or virtual) or in my experience individuals working in the same region are more than happy to share their experiences. For evaluation specific education or new methods (even seasoned evaluators need some inspiration every once in a while!) consider online resources, like EvalAcademy.
These strategies were summarized with LRS evaluations in mind, but I am confident they can be adopted for evaluation projects in all settings. If you have any other resources or strategies for evaluating in a LRS, please comment below.
Resource
Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health. 2006;60(10):854–857. doi:10.1136/jech.2004.028662