State of Evaluation

Measurement and Evaluation Practices in 91ÊÓƵ’s Nonprofit Sector

91ÊÓƵ’s nonprofit sector is made up of tens of thousands of organizations driven to build thriving communities and a dynamic province. They bring people together to address the social, cultural, and economic issues that act as obstacles to this vision. To succeed, public benefit organizations must be capable of responding to changing circumstances quickly and redirecting resources where they can help the most people or have the greatest benefit. This is why evaluation work is so important to the nonprofit sector.
Though it is important, evaluation work isn’t always easy. It can be time consuming, expensive, technically difficult and stressful. Over the last several years, we have sought to better understand how evaluation is practiced by 91ÊÓƵ’s public benefit nonprofits, and to identify strategies to make evaluation more efficient, more meaningful and more action oriented. We have learned that evaluation is often most useful when it is designed from the ground up in a collaborative way that creates safe spaces for discussion and informs deep learning. We have learned that there are many different approaches to evaluation, each useful under the right circumstances (and much less useful under the wrong circumstances). We have learned that there is quite a lot of evaluation work going on, but insufficient discussion of the intended purpose of this work. We don’t talk enough about whether evaluation efforts are delivering on their promise to create a sector that is more informed, more responsive, and more impactful.

Our discussion about how to improve evaluation practice has been limited, to date, by the lack of detailed information about current practice. We know nonprofits spend significant time and resources on evaluation, but we have not known exactly how much. While we know that some evaluation projects are much more useful than others, we have not known what proportion of evaluation reports get used across the province, by whom, or for what purpose. While we know that the evaluation function is often under-resourced, we have not known how often this is true, who pays for evaluation work, or who most commonly carries it out. The fact that we do not have a clear picture of how evaluation is practiced across the province may itself be an indication that our sector has not been as strategic as it needs to be in approaching this essential component of our work. Together, all of these issues have contributed to a dynamic that has sometimes led to an emphasis on accountability over learning and a missed opportunity to strengthen relationships across the sector and communities.

For these reasons, we are very excited to share with you the first-ever ‘State of Evaluation’ report for 91ÊÓƵ. This is the first detailed picture of how evaluation is practiced across our province, and how the findings of evaluation work are being translated into action. Our sincere hope is that this report sparks a rich discussion about improving evaluation practice and develop a strong, responsive, impactful nonprofit sector.

Read the report: State of Evaluation

French version: L’Évaluation : État des lieux

Imagine a grantmaker that has very clear investment priorities. Perhaps, for example, their focus is on helping young adults transition from school to employment, and they seek out grant recipients capable of making a measurable impact on job search skills, career planning skills, and connections between students and employers.

This grantmaker may receive proposals from a number of different organizations, each with equally strong potential to achieve measurable change in one or more of these outcomes. However, these proposals may vary widely in their potential to generate useful learning for this particular grantmaker at this point in time. One may be scaling up a tried-and-true model, while another may be proposing something highly experimental. One may have strong connections to private sector employers, while another may have great relationships with the local college or trade school. One program may be delivered by established professionals, while another intends to have the program designed and guided by an advisory committee made up of recent graduates.

None of these differences are bad. However, depending on the organizations involved and their context, some of these scenarios may have greater potential than others to generate useful, actionable learning. This is why we’ve been exploring ways to help grantmakers and nonprofits seek out alignment in learning goals and processes as well as alignment in intended outcomes. One of the biggest things we have learned is very simple. At the very beginning of the relationship — at the grant application stage — people should talk about learning before they talk about measurement. In Achieving Greater Impact by Starting with Learning, we explore ways in which funders and nonprofits can begin a relationship that is defined by shared learning at the grant application stage.

In our last blog post, we talked about what it means to be a learning organization. Organizations that have a strong learning culture are intentional about learning, and they recognize that listening to their partners is essential. If we are serious about making better use of information, being responsive to the needs of service users or clients, and creating better feedback loops then we need to have a discussion about the types of relationships that get prioritized between funders and nonprofits.

When funders ask the organizations they fund about their evaluation plans, they are typically motivated by a desire to achieve the greatest impact possible through their investment. They often hope to help the organizations they fund to do the same. However, these conversations sometimes veer off track, especially when nonprofits feel pressure to produce evaluation results that align with funders’ preconceived ideas. Evaluation can turn into a tool for accountability and risk management rather than a tool for learning. Unfortunately, conversations that focus on learning — if they happen at all — tend to take place too late in the process and might only show up as one question among many in a report somewhere along the way.

As notes:

The constant pressure for NGOs to demonstrate results generates an understandable concern about publicising or even sharing lessons learned from programme experience. The reluctance to be open about learning may be particularly strong where a programme has not achieved what was promised in funding applications for fear of the repercussions that may result.

Learning is so important to the work that is going on in the sector, that it needs to be baked into relationships from the start. We think the grant application process could be designed in a way that is less risk-averse, creates trust, and allows for more honest conversations. The graphic below illustrates the difference between the two types of conversations that can emerge between nonprofits and their funders.

As part of our research, we’ve identified five principles to help promote a learning relationship (plus one other just for funders that can be found in the full report):

• Model a culture of learning. Funders and nonprofits should demonstrate their own commitment to learning from the process of communication by sharing information about their own learning cultures.
• Learn in partnership. Learning requires reaching outside one’s comfort zone and listening to others. It requires developing a different kind of relationship. It also requires respect for the fact that your partner organizations may learn in different ways than you do and face different constraints to their learning.
• Plan for learning. Identifying up front what each party hopes to learn is key. While acknowledging that learning is, by nature, iterative and can include unexpected results, planning sets the stage for more honest conversations going forward. It can help form trust between the parties and clarify alignment.
• Reward learning. Make it clear from the beginning that a commitment to learning will be valued in the decision-making process and in managing the grant on an ongoing basis. This can also help to reduce the reporting burden on potential and future grant recipients and encourage discussion of what really matters.
• Be flexible and fair. Since culture is a dynamic phenomenon and learning often arises in unexpected ways, cultivating a learning culture necessarily involves surrendering some control over the process and acknowledging mistakes. It is important that all stakeholders are comfortable with this shift and that they are supported to manage it.

For funders, engaging with applicants and grant recipients in this way requires a willingness to share control over the planning process and the learning agenda, and this can lead to uncertainty and risk. It also requires unlearning some bad habits (such as treating measurement and evaluation as accountability mechanisms rather than opportunities for shared learning).

For nonprofits, it means going back to those ideas we mentioned in our last blog and may require potentially making some adjustments to get comfortable working in a way that builds feedback loops, creates more opportunities for sharing with others, and promotes reflection.

In short, it can take time to understand the complexities and nuances of a learning culture, and to develop the new skills required to create and sustain learning relationships. Working in this new way isn’t always easy and may require patience and iteration. However, the potential pay off is ultimately what we all should strive for: Better understanding of the issues, stronger relationships, and greater impact.

Read the report

Andrew Taylor is the co-owner of Taylor Newberry Consulting, which specializes in developing research and evaluation solutions for public sector organizations. He is also ONN’s Resident Evaluation Expert. Ben Liadsky was ONN’s Evaluation Program Associate and project lead for the Sector Driven Evaluation Strategy from 2015 to 2017.