We’ve defined and written about How to Write Good Evaluation Questions before. We’ve even shared some examples of evaluation questions based on the sector or content area. But there is another way to think about evaluation questions. Let’s look at how using different evaluation strategies or frameworks can help you to craft those perfect evaluation questions.

To describe how evaluation questions differ based on type of evaluation, let’s make up a hypothetical program to evaluate.

You’ve been hired to evaluate a program that has been operating for one year. The program aims to engage youth in an after-school program with the goal of keeping at-risk youth safe. They offer physical activity programs (e.g., basketball), tutoring, and life-skills programs (e.g., cooking classes), as well as lounge areas.

Scenario 1: Formative Evaluation

The purpose of formative evaluation is to assess how the program is being implemented. Key evaluation questions should focus on enablers and barriers of implementation. For example:

Awareness

Are youth aware of the program?
Are families aware of the program?

Utilization

What are program utilization rates?
Why are youth attending, or not?
What barriers exist in attending various programs?
What parts of the program are most desirable?

Implementation

What is working well?
What could be improved?

I’ve found that looking to the field of implementation science can help to outline themes and even specific questions that might be relevant to formative evaluation.

Scenario 2: Summative Evaluation

After a year of operations, the program leaders believe it is time to assess the impact of the program. Your summative key evaluation questions should focus on outcomes:

Utilization

Who used the program? What is the profile of the youth who attended our program?

Outcomes

Did more youth graduate high school? Did fewer youth drop out of school?

Did youth experience fewer disciplinary actions at school? In the community?
Did youth learn new life skills? Do those life skills support them in gaining employment?

Scenario 3: Using the RE-AIM framework

First a quick note: most of these scenarios are not mutually exclusive. RE-AIM is well suited to both formative and summative evaluation. For an overview of RE-AIM read this.

RE-AIM essentially comes with its own questions, that only need to be adapted to your specific program. The key evaluation questions are based on the 5 focus areas of RE-AIM:

Reach – Who is coming to the program? Who is not?
Effectiveness – Is the program engaging at-risk youth? Are more youth finishing high school?
Adoption – How does the program collaborate and integrate within the community? What partnerships enable the program’s success? How is awareness being spread?
Implementation – How is the program being implemented? Is it implemented as intended? What barriers exist?
Maintenance – Are the outcomes sustained over time?

Scenario 4: Developmental Evaluation

The program is interested in really understanding how they can be adaptable to the current context and how they can continue to develop and grow to have a sustainable impact on the complexity presented in at-risk youth. In the last year they haven’t yet landed on a stable program – constantly adapting to new information and changing environments. They want to use Developmental Evaluation to support rapid growth and emergent innovations. Key evaluation questions should focus on emergent learning and opportunities for development:

What are we learning that informs our development?
How are we engaging the community?
What evidence of effectiveness is useful to our development?
What opportunities are emerging?

For more great examples of DE evaluation questions, check out Developmental Evaluation Exemplars: Principles in Practice edited by Michael Quinn Patton, Kate McKegg and Nan Wehipeihana.

Scenario 5: Utilization-focused evaluation

Again, Utilization-focused evaluation is not at all mutually exclusive from previously described scenarios. In this scenario, you determine that the primary purpose of evaluation is to give the staff something they can action (or use). In this case, the approach to development of the evaluation questions is key: engage your stakeholders often, early, and deliberately. What do they need to know? What decisions do they need to make? Key evaluation questions may include:

How can the program attract more at-risk youth?
How can the program retain/continue to engage at-risk youth?
Which programs have the most demand?
Which programs have the best return on investment?

Scenario 6: Most Significant Change Evaluation

Most Significant Change evaluation moves away from standard indicators and measuring. It uses storytelling to evaluate the success of a program through the lens of various stakeholders. For this evaluative approach questions may be tailored to each audience:

For program staff:

What is the biggest success you have seen?

For youth:

What is the biggest change you have experienced since participating in this program?

Scenario 7: Outcome Harvesting

Outcome Harvesting is a participatory evaluation methodology. Instead of key evaluation questions, Outcome Harvesting focuses on Outcome Descriptions. You may ask questions of the stakeholders like:

What have you worked on? (or participated in?)
What was the significance of the work you did?
What impact do you think it had?

Scenario 8: Principles-Focused Evaluation

Principles-focussed evaluation aims to assess how an applied set of principles translated into action or behaviour. Your methodologies may be very similar to other evaluation approaches, but the questions may centre on a pre-defined list of principles. Let’s say the program had a list of core values that included:

Person-centred supports. The program aims to offer support to each youth, based on individual, specific needs.
Welcoming. The program has no exclusionary criteria.
Fun! The program encourages staff and youth to have fun and try new things.

Your evaluation questions may then look like:

Do youth feel supported?
What barriers exist to accessing the program?
What is the experience of participating?

Again, Michael Quinn Patton has an entire guide to Principles-Focussed Evaluation that can help you develop the right evaluation questions.

Hopefully, this has helped to showcase how some evaluation approaches and frameworks can shape your evaluation questions.

Program evaluation is certainly not limited to one dominant question. Most evaluations can assess several domains. The rule of thumb is 3 – 5 key evaluation questions, with nuanced sub-questions embedded if needed.

Evaluation Question Examples by Type of Evaluation