This article is rated as:

A real-world example

I am of settler-Métis descent and a member of the Métis Nation of Alberta. Currently, I live in Medicine Hat which falls in Treaty 7 and Treaty 4 territory. My family comes from the Battlefords in northern Saskatchewan, but I mostly grew up in Calgary. I worked in non-profit organizations in Calgary and Medicine Hat, primarily in roles that focused on operations, data management, and reporting. I currently work as an evaluation assistant at Three Hive Consulting. I also volunteer within my community.

The organization I volunteer with received funds to host a round dance in our local community. The round dance was open to the public. I thought it would be a good opportunity to share with you how we evaluated this event and reported back to the funder.

This non-profit exists for Indigenous women and two-spirit people to both learn and teach Indigenous culture. In everything we do, we look for ways to apply Indigenous ways of knowing, being, and doing. So, how do we evaluate the outcomes and successes of an Indigenous ceremony using an Indigenous worldview, while honouring protocol and oral tradition?

 Most of us involved in evaluation have commonly used these methods:

• counts of attendees,

• satisfaction surveys, and

• the occasional pre- and post-event surveys around knowledge gained.

These methods didn’t feel like they would tell the full story or be the right tool for this evaluation. From an Indigenous worldview, the evaluation should tell the whole story. So, what is the whole story? Where does the story of this ceremony begin?

Where did we start?

Another volunteer and I met to discuss ways to evaluate this Indigenous ceremony. What was it that we wanted to know? Of course, we wanted to report on the funds in a meaningful way, but how could this evaluation also help us as an organization?

 We identified the main questions we wanted to answer:

• How can we improve our planning and preparation for ceremonies and craft circles?

• What impact do these ceremonies and craft circles have on our community?

• What other gatherings would our Indigenous community like to see?

Our questions involved two different levels of community. When it comes to impact, we not only want to get the impact on the Indigenous community but all who participate. However, when it comes to planning future activities, we want this information from our Indigenous community only.

Considerations

When we were planning our evaluation, we did not include the pipe ceremony or its accompanying feast that were held before the round dance. We didn’t feel evaluating these ceremonies was appropriate due to their sacredness. A pipe ceremony tends to remain more in oral learning and not in writing. However, we did note in our report to the funder that these ceremonies were held on the same day as the round dance.

We focused our evaluation on what participants got out of the ceremony and how we could improve our preparations for implementing the round dance. We did not centre any of the evaluation on improving the ceremony itself due to the traditions and protocols that need to be followed. Our organization can change how we plan or prepare for a ceremony, but we wouldn’t make changes to the ceremony itself.

What are we evaluating?

Round dance requires extensive preparation. It is part event planning and part cultural teachings. Not everyone in our community had the opportunity to learn their culture, so every time a ceremony is planned, teachings must be given in preparation. In addition, we needed to prepare some ribbon skirt craft circles as this was the regalia required to be worn for the round dance.

On average, in a ribbon skirt craft circle, it takes about 5 hours for someone to make their first skirt, and each participant needs access to their own sewing machine. The space required for this and the number of people who needed access to these teachings meant our members needed to run three ribbon skirt workshops within a month.

So now our evaluation will need to include ribbon skirt craft circles and the round dance ceremony for which we were funded.

How we conducted our evaluation

We knew that the evaluation would need to be done by our volunteers, so we needed to keep the data collection fairly simple.

1. Quantitative data

   • We counted the number of people who attended the ribbon skirt craft circle. We commonly count attendees at our craft circles, so this would be easy to gather. Reflecting back now, it would have been useful to also count how many skirts were made, as that metric would inform how we might run a ribbon skirt making differently. We did find that not everyone was able to complete their skirt, so it would have been helpful to have a count of how many were completed, how many were incomplete, and how many people took kits to finish at home as they were busy helping others.

   • We counted attendees at the round dance. Students volunteered at the round dance, so the count of attendees became their metric to track. As this is an annual event, we find this measure useful to see if we have growth year over year. We also use this count to assess if the space we are using is big enough.

2. Qualitative reflections

   • Through informal interviews during one of the craft circles and at the round dance, we were able to gather attendee feedback on what they gained through their participation. Those who participated were informed that the information shared would improve our service offerings and help us report to funders. We also collected participant feedback to show the impact of holding a round dance in our community.

   • After the ribbon skirt circle, I spoke with one of the facilitators about how it went from their perspective. It was recommended that when teaching people to make their first ribbon skirt it is probably best to stick to a simpler skirt pattern, as pockets were a little too hard for those with less sewing experience and a simpler pattern may have increased the number of people who were able to complete their skirts in the craft circle.

   • We were also able to gather group feedback at a members’ meeting shortly after the round dance. This group was asked to provide their reflections and feedback about the round dance. I opened this conversation broadly with, “Does anyone have thoughts or feedback to share about the round dance?” I commonly ask for thoughts and feedback from this group after we host events or anything that requires major planning. So, they are used to me asking a similar question and are ready to give their thoughts on what worked well and where we could improve. I did probe a bit more about what their highlights were from the round dance. Some of this feedback gets recorded in our meeting notes and I also take notes to supplement the meeting notes.

Our qualitative data gathering was informal. We didn’t use formal guides or scripts. Our volunteers were given a couple of questions they could use in these conversations. We aimed to use a conversation style, and much of it was based on familiarity or relationship building.

All these data sources formed the basis of the evaluation of the round dance. The quantitative data and qualitative reflections gathered would be used to report back to our funder. We would also use these evaluation findings to inform improvements for the craft circle experience and annual ceremony planning.

Given this is a new and growing organization and this funder was the largest single funding source so far, any reporting would be new to its membership. 

Lessons learned

Some of the lessons we learned through this evaluation were:

• Discussions with participants are time-consuming. Each conversation takes time because relationship-building is the priority. Even as an active member of this community, we first need to catch up on everything since we last spoke. Because everything is about relationships, I can’t just start asking questions about participation. Relationships are key to meaningful conversations.

• Ensure there are more volunteers available to talk to participants about their experiences and keep their goal for the number of conversations low. We were too ambitious for how many discussions we thought we could have due to both the time these conversations take and the desire for all who are at the round dance to be able to participate in a conversation.

• Prior to the ceremony and craft circles, gather the volunteers together who are going to have these discussions to practice working the questions into conversations. Most of the volunteers had difficulty or felt uncomfortable asking participants about their experiences. I think it would have been helpful to have the volunteers do some role-playing to gain confidence.

• It would have been better to focus on collecting participant feedback immediately after the ceremony. I had more meaningful conversations when we were cleaning up and people were starting to head home, as people were more eager to share their experiences and feedback was easier to hear. Carrying a pen and paper around for the entire ceremony is not practical and would hinder my participation in the round dance. By only gathering feedback as we were wrapping up, simplified how long I had to keep these supplies on me. I made notes after conversations as recording in a loud gymnasium would not have provided useful recordings.

• Discussions after the round dance were just as important. Some people need to be asked a couple of days later to get some richer responses. These later conversations were where discussions about not only what people got out of their participation came through, but also their ideas as to how to better prepare for the next round dance.

• Allow space for the unexpected. Through some of the conversations I was able to have, I learned about people’s thoughts on the non-profit’s other programs and the strategic plan. I was not expecting that type of feedback from these conversations, but that input may help lead future discussions for members about the organization's future directions.

For future evaluations, I would like to involve other volunteers sooner in the planning and preparation of the evaluation. Their earlier involvement would help them to understand how we developed our evaluation questions to support buy-in and let them practice the skills required to lessen any uneasiness in our data collection methods. For a future round dance, I would concentrate on connecting with participants immediately after the ceremony. This might not apply to all ceremonies, but it was suitable for this one as this was a healing ceremony, most participants are energized after. It was also beneficial to gather additional feedback a few days later, as this time allowed for some participants to further reflect on their experience.