What program managers need to share with their new evaluators
This article is rated as:
Evaluators – this one’s not for you, but maybe you can share it with some clients!
As a program manager or project lead, you may find yourself needing an evaluator. How do you get that evaluator oriented to the project? What information do they need? Let’s walk through it.
Hopefully your new evaluator has a few questions to kick things off. Usually things like “What is the purpose for needing an evaluation? How will the information be used? What do you need answers to?” You might even sit down for an engagement meeting to answer most of those questions. This is my job. I do this on repeat. And yet no matter how I try to harvest all the important information from the key stakeholder group, I still find that I’m often missing core pieces of information when I set out to develop my evaluation plan. Key information tends to trickle out in bits and pieces over the initial weeks or even months. So, I thought it might be helpful to compile a list of what makes the new evaluator orientation and planning process as efficient and effective as possible.
Here's a checklist of things every program manager should share with (or tell) their evaluator:
Describe the program.
This sounds obvious, but you’d be surprised how often the people that hire me don’t start with a clear overview of what the program is! Be prepared to describe, from start to finish, what it is you do or what service you offer and who you serve.
TIP: Share any documentation you have, including project proposals, grant applications, or work plans. Include a description of who’s who in the zoo: what are the key roles.
Has the program been evaluated before?
If yes, share previous:
Evaluation plans
Evaluation reports
Data (if you want year-to-year comparison, for example)
Logic models or Theories of Change
TIP: It can be helpful to share what you liked or didn’t like about your previous evaluations and reports – what missed the mark? What was really helpful? What do you want this new evaluator to do differently or the same?
A description of current data collection processes.
Ideally this would be documented but I’ve yet to find a program that has this (and have found myself creating process maps myself for the complicated programs). In the very least, be prepared to share a verbal description of how things work. For example:
“When a participant expresses interest in our program, they sign a consent form (here’s a copy) and fill out a baseline survey (here it is). Then they attend a weekly workshop for 2 months. We keep an attendance list (like this). Then there is a post workshop session survey (here it is).”
Note that sharing the current data collection tools is critical.
TIP: It can be helpful to share what’s working or not working about data collection. For example: “We really struggle to get a decent response rate on our post workshop survey.” Or “We collect those attendance sheets, but we have no use for them, they just get recycled!” Even detailed insights can help “On our survey, the age ranges we offer aren’t a great reflection of our clients.” or high-level reflections “We have no ability to match data between two very important data sets”.
Funder and reporting requirements and timelines.
Do you have mandatory reports due? Even if you don’t intend to have the evaluator write them for you, be clear about who is contributing to what and what the timelines are. Often the evaluator can time data collection to be ready for your annual, quarterly or interim reports. They can certainly help you out by ensuring that key metrics or key success factors are captured and ready for you. So even if writing the report(s) is beyond the scope of the role you want the evaluator to play, it can be helpful to share these anyway.
TIP: Documenting who is responsible for what in mandatory reporting is very helpful. You could ask your evaluator to contribute to certain sections, contribute data, or plan to use the evaluation report as an appendix to the required reporting template.
Who are the decision-makers?
Is there a program steering committee or advisory group? Perhaps there is even an evaluation advisory committee that is disbanded or defunct? Share these details with your evaluator so (a) they can ensure their reporting is prepared for the appropriate audience(s) and (b) they know who to go to for important decisions – like signing off on the evaluation plan.
TIP: I’ve found it to be effective when a program has a core group of no more than 5 people who provide input into an evaluation. Five seems to be able capture enough diverse perspectives, but with more than this it gets hard to convene for key discussions (and timelines get pushed out).
BONUS TIP: Be prepared to answer key evaluator questions.
I get that it can be difficult to come up with great insights on-the-spot – so don’t! If you have an evaluation meeting coming up, try to carve out even 15 – 30 minutes in your schedule to prepare for it.
Be prepared to describe why you want this evaluation, what questions you have about your program and how you plan to use the information.
Consider reviewing your program material – work plans, proposals, previous evaluation reports, even meeting minutes. Were there key questions posed by program stakeholders or partners? Take a few minutes to reflect: If I could know anything, what would I want to know about this program? It could be about the way it’s run, about the impact it has, or about the effectiveness or efficiency of it.
Evaluators want to provide you with information that is valuable, relevant, and actionable! Setting them up with the right information makes it more likely that you will find value in the role.