Criteria Based Ranking in Developmental Evaluation
This article is rated as:
Developmental Evaluation is widely implemented and the preferred option for programs that address complex problems such as poverty and homelessness. There is a growing body of literature on Developmental Evaluation (DE) and more and more evaluators are embracing this approach (myself included!).
For the past six months, my colleagues and I have been involved in DE. We have helped a client make important decisions using multiple evaluation tools, including surveys, document reviews, and Criteria Based Ranking (CBR).
In this article, I will explain what Criteria Based Ranking is and how we used it in Developmental Evaluation.
What is Criteria Based Ranking?
CBR is a much simpler form of Multiple-Criteria Decision Approach, which comes from operational research, a discipline that deals with the development and application of advanced analytical methods to improve decision-making.
Both CBR and Multiple-Criteria Decision Approach evaluate multiple and often conflicting options, such as cost versus quality. For example, in a publicly funded healthcare system, when comparing the benefits of a new drug to the status quo, decision makers need to weigh the health benefits and economic impact of both options. It is difficult to compare cost versus effectiveness directly. CBR allows us to have a final numerical number (rank), while accounting for both criteria.
For the DE project, we are supporting a team of community leaders that aim to improve services for seniors within the city. The decision they were confronted with was: out of the many problems and challenges seniors face, which ones should the project prioritize and use in their engagement strategy? To address this question, we first completed a document and literature review and identified 20 priority areas.
Next, we helped them further narrow down the priority areas using a simple form of CBR.
How to use Criteria Based Ranking in Developmental Evaluation
In CBR, the first step is determining relevant criteria you would like to use. In the healthcare example above, impact on health status (i.e., survival rate and quality of life) and on health care cost (i.e., cost of drug and estimated cost-savings of future healthcare cost) can be used to compare the new and the status quo drugs, and arrive at a final score.
Next comes assigning a value for each criterion. For our DE project, the criteria the stakeholders picked were equity, feasibility, urgency, and potential for joint action and they decided that each of the criteria were equally important so were given the same weight. However, depending on the priority of the DE project, higher value can be assigned to some criteria if they are determined to be of higher importance relative to the other criteria.
The final step in CBR is assigning numerical values to determine a rank. At this stage participants individually rate each of priority areas using the different criteria.
For our DE project, using an online survey platform, we asked the stakeholders to rate the 20 priority areas from low to high using the four criteria shown below.
We completed the analysis, and the results were surprising. Out of the 20 priority areas, the ones that stakeholders felt were sure to top the list were ranked lower and vice-versa. These results show that topic areas that often grab attention are not necessarily the ones that will be prioritized when using common principles (criteria).