Strategic Learning and Evaluation – What Boards Need to Know
This article is rated as:
Recently I was asked by a client about an evaluation literacy course for its board. The client’s board members had just attended a strategic planning day and through that discussion felt they needed education on evaluation and metrics. On one hand I thought “bravo, they want to know more about evaluation!”; on the other hand I thought “shit…., I’ve totally failed them as their evaluator – what have I been missing?”
Boards need quality information to make strategy and leadership decisions; however, the reality is this board isn’t getting the information it needs to inform its decisions. As their evaluator, it is my responsibility (and also opportunity) to show them the way forward, so they are no longer left with answers that are “a definite ‘maybe,’” but instead have data and insights that they can use to inform their decision making. This means a more systematic, coordinated, and intentional approach to evaluation and learning – a strategic learning and evaluation system (SLES), as is described in FSG’s Building a Strategic Learning and Evaluation System for your Organization. The course I developed therefore focused not only on evaluation literacy, but also how evaluation can support a SLES. The course has three overall objectives for its board members:
Understand the necessity of and advocate for strategic learning;
Understand the basics of evaluation and how it can support strategic learning, and;
Begin developing the building blocks for a strategic learning and evaluation system.
Here are some of the key learnings and action imperatives for the board:
Understand that evaluation is one piece of the learning pie
While evaluation is important for learning and improvement, it is only one of many information-gathering approaches that can be used to inform decision making about strategy. Organizations also collect information through performance measurement, audits, research, case studies, discussions at the water cooler and a number of other ways. So, if the ultimate goal is for organizations to learn and use that information to improve, boards need to shift thinking from “leading with evaluation to leading with learning” (Centre for Evaluation Innovation).
Evaluation is one of many ways to gather information to inform decision-making and learning. Others include performance measures, monitoring, audit, research, cost effectiveness analysis, and case studies.
Understand and advocate for strategic learning
I am lucky that my client’s board has a desire to learn more about evaluation and how it can support strategy development. The Centre for Evaluation Innovation recently conducted a survey to collect data on evaluation and learning practices in foundations and found that senior management often communicate support for evaluation, but their behaviours do not demonstrate support for evaluation. So what are the board behaviours that would demonstrate support for evaluation? And even more specifically, evaluation for strategic learning?
The Centre for Evaluation Innovation produced a report titled, Evaluation to Support Strategic Learning: Principles and Practices, in this report they explain that “designing data collection and evaluation specifically to support strategy decisions requires shifts in thinking about what questions get asked, the role the evaluator plays, how data collection is timed, and the framing of the findings” (pg. 3). They go on to articulate nine principles of evaluation for strategic learning that boards can advocate for within their organization:
A support for strategy
Integrated and conducted in partnership
Emphasizes context
Client focused
Places high value on use, and helps to support it
Data to inform strategy can come from a wide variety of sources and methods
Must take place within a culture that encourages risk taking, learning and adaptation
Is flexible and timely and ready for the unexpected
Is constructivist
Understand the basic evaluation terms and steps
Part of advocating for strategic learning is understanding basic evaluation concepts and terms. If learning and evaluation efforts are to inform an organization’s decision-making practices, then boards need a clear vision for evaluation – what it is and is not.
I’m not going to lie – I don’t believe in reinventing the wheel, and Chris Lovato and Kylie Hutchinson put together an Evaluation for Leaders course. Much of what I covered in the evaluation basics module of my course follows what they outline for evaluation terms, types and steps. However, as Chris and Kylie say in their course:
“Leaders and decision makers don’t need to be evaluation experts, just expert supporters and users.”
So, in my course there is an emphasis on the first evaluation step – focus. For boards it is important to understand this step. An evaluation can’t be all things to all people – focusing it provides a clear direction of who needs what information and how the user(s) are going to use it.
Understand what is credible evaluation evidence
Thinking has changed on what constitutes “credible” evaluation evidence. Many board members from this organization come from a science background that believes RCTs (randomized control trials) are the gold standard for evaluation evidence. This is a common misconception, but one that is particularly important to address with board members. As the Evaluation Yoda, Michael Quinn Patton states:
“Despite the recognition more than 35 years ago that the reductionist approach to complex problems is likely to fail, many still persist in believing that we must rigorously apply the scientific method to problems in medicine and public health.”
If a board is trying to impact systems and trying to shift the conditions that are holding problems in place (i.e. systems change) then it is important that the board shift their thinking from measuring and proving against some sort of fixed model to understanding and improving – in other words, to adopt a systems thinking lens. The evaluation methods that are selected to evaluate that change comes back to how appropriate the methods are given the purpose of the evaluation, the questions it needs to answer, and how technically adequate the findings are given the time and cost constraints. As the United States General Accounting Office Program Evaluation and Methodology Division (1991, pg. 17) quotes:
“A strong study is technically adequate and useful – in short, it is high quality”
Know what you want/need and communicate it to your organization
If a board is not clear on what information it needs to inform its decisions, you can be sure the rest of the organization won’t either. A board gets a lot of information, but it may not:
Contain the right information,
Be presented in a useful and useable format,
Be received on time (i.e. after a decision has been made), or;
Be connected to organizational strategy, which means the findings aren’t getting used (or at least not fully).
An easy first step to enhance use of evaluation findings is for boards to make their timelines and reporting preferences known. A more difficult next step a board should consider when trying to enhance usability of findings is to implement a SLES.
According to FSG’s Building a Strategic Learning and Evaluation System for your Organization, a SLES contains:
A clear vision for evaluation,
A culture that fosters individual, group and organizational learning,
A compelling and strong strategy,
Coordinated evaluation and learning activities, and;
A supportive environment.
Bottom line – a SLES will provide guidance and align organizations on who, what, when, where, why and how to measure and report.
I left my client a lot to chew on. As I mentioned above, implementing a SLES will be difficult, but ultimately should provide the board with an evaluation strategy that increases the value of evaluation for its organization.
Interested in learning more? Sign up for our newsletter and we’ll notify you when the online version of this course is available.