What is Evaluation? A Review of AEA’s Recent Post
This article is rated as:
Evaluators are equipped to ask and answer some pretty complex questions. We are not afraid to tackle abstract ideas and make sense of messy data. However, there is one question that can be tough to succinctly answer - what is evaluation?
As evaluators, you can likely recall sitting at family dinners or catching up with a friend when they ask you "what is evaluation?" You may have even had this conversation with colleagues or clients. My initial reaction is "where do I begin?" or "how do I explain evaluation to a non-evaluator?"
Many evaluators and evaluation societies have attempted to answer this question. The Canadian Evaluation Society (CES) consulted members and published their findings on this very topic. This is the definition they came up with:
Evaluation is the systematic assessment of the design, implementation or results of an initiative for the purposes of learning or decision-making.
The need to explain evaluation is likely to come up again and again for evaluators. Fortunately, the American Evaluation Association (AEA) recently created a series of videos and short summaries to explain evaluation. Their post is intended to help us define our profession and better explain the value of evaluation to outsiders.
AEA presents evaluation in the context of three sectors: Organizational, Teaching, and Independent or Consultant. To better understand the work of evaluation in the sectors, AEA interviewed three evaluators. The remainder of this article will review learnings from the AEA post as well as some potential implications for the future.
Evaluation Sectors
Organizational Evaluation
This sector refers to working internally within an organization. Some examples may include: government, non-profits, charities, foundations, or even industry. AEA interviewed Eric Barela, the first Measurement & Evaluation professional hired by Salesforce. Here is a summary of what he had to say about being an internal evaluator:
Approach evaluation by asking three questions:
What? To gain an understanding of what an organization is doing
So what? To see if anything is happening
Now what? To use the information to make change or move from "good" to "great"
Evaluation is becoming more applicable to other industries, including for-profit organizations. He feels this is driven by the desire to focus more on social impact.
As an internal evaluator, he helps his organization ask questions, understand data, and then act on it. He poses two other questions that help him in his evaluation role at Salesforce:
How do we know this is happening?
What do we need to know in order to improve what we’re doing?
Teaching Evaluation
The teaching of evaluation within academic settings is viewed by AEA as a main sector of practice. AEA recognizes that evaluators come from diverse backgrounds and there is currently no set educational pathway nor is there a licensing body for the profession. However, there are academic institutions that offer evaluation courses to help students prepare for a career in evaluation.
Karen Jackson, from the Leadership Studies and Adult Education department at North Carolina A&T State University, was interviewed to gain more insight into the teaching of evaluation. Here are some key takeaways from her interview:
Defining evaluation is a complicated task, but the definition should be flexible to allow for adapting in diverse settings.
Academic institution, like hers, offer evaluation courses that focus on the methodology, cultural competency and application of skills.
She often starts a course by asking her students "what is evaluation?" The initial perspective is that evaluation is a means of assessment, testing, or a way to meet funder requirements.
As her courses progress she encourages her students to ask themselves "why they are doing an evaluation." This leads to answers that include learning, improvement and trust – which she argues are at the root of evaluation.
Independent Evaluator or Consultant
The final sector described by AEA refers to external evaluations that are conducted by independent evaluators or consultants. AEA views independent evaluation as a way to boost public confidence or verify what an organization is doing. AEA interviewed two consultants about their experience in independent evaluation: Jara Dean-Coffey and Karen Jackson. Both interviewees aim to discover insights to help their clients make decisions. Here is a summary of what they had to say:
Jara Dean-Coffey
Relies on six areas to help her clients:
1. Evaluative thinking
2. Evaluative practices
3. Strategy
4. Communication
5. Capacity building
6. Coaching
She recommends always starting with evaluative thinking to understand: "the context, the assumptions, the values, and the outcomes" of her client. This includes reviewing the organization’s theory of change.
One strategy she offers is to start with what the client feels is most important for them to answer at the moment.
Karen Jackson
Karen Jackson views her purpose as "help[ing] organizations learn more about themselves."
She seeks to understand if clients are achieving their goals and if not, help them take the steps to do so.
She believes that independent evaluators have the opportunity to give people a voice and work with communities.
Next Steps
The breakdown of the definition by sector combined with the three evaluator perspectives was a helpful way for AEA to explain evaluation. There were also common themes that emerged across the sectors, such as: learning, helping others to ask meaningful questions, and identifying ways to improve. Revisiting Eric Barela’s use of the question "now what?", I have some suggestions on how to use the AEA post and insight moving forward.
Ask yourself the same questions to understand your practice: What does evaluation mean to you? How do you approach evaluation? And what do you find rewarding about your work?
Look for sectors that may be missing from AEA’s overview. For example, should ‘Evaluation Research’ be included in the definition?
Create your own definition of evaluation. Using AEA’s recent post and/or other definitions (such as the one offered by CES), create your own definition of evaluation and what you do in your role.
Advocate for evaluation in these three sectors and beyond. Large for-profit companies, like Salesforce, are starting to see the importance of evaluation – let’s keep that momentum going!
Conclusion
AEA’s post does a good job at explaining evaluation in a succinct way. For me, I have never worked in the Learning Evaluation sector, but I still found Karen Jackson’s approach and perspective on evaluation to be both relatable and useful. Likewise, I think movement between these sectors is expected – both in terms of the definitions put forward, the skillsets used and the nature of evaluators having the ability to work in multiple sectors.
AEA’s explanation of evaluation is similar to many other definitions put forward by societies (i.e CES). I think what differentiates this post from other definitions is that it incorporates the experiences and perspectives of multiple evaluators by effectively using video. The use of interviews result in an overview of evaluation that is meaningful to evaluators as well as outsiders.
I think the question "what is evaluation?" and our attempt to define it is an on-going process with no end in sight. I think this is what makes evaluation interesting – it continues to expand and evolve by incorporating new ideas and entering new settings. A flexible definition is key to our profession and I think AEA’s post does a good job at showing the complexity, yet inclusivity of evaluation.