One of the best feelings in the world is the ‘aha moment.’ It’s a wonderful moment of intense clarity where all the pieces of the puzzle suddenly fit together and everything just makes sense. Sometimes it takes a little longer to reach these incredible moments of discovery. During a recent workshop, one of the biggest ‘aha moments’ that participants shared with us in their post-session feedback is the realization that they were dead wrong about evaluation.
This was a real game-changer because up to that point, these participants thought they had been doing evaluation of their programs. They realized that what they had really been doing was actually program monitoring. Program monitoring is a process that helps improve performance and achieve results.
Activities such as consistently recording the number of program participants or collecting post-program surveys are valid and responsible ways to monitor program performance, but they are not on their own considered evaluation. It was thought that by measuring in this way and then responding by making tactical or strategic adjustments to their delivery, that this enabled them to check the ‘evaluation’ box for their programs and services.
There is nothing wrong with this approach. Monitoring and management are important parts of being responsible stewards for the programs and services we deliver in our community. We know that even funders sometimes use the term evaluation, when what they’ve asked for in reality is a sound plan to measure and report on program activities.
Let’s have a further look at evaluation and monitoring.
So, what exactly is evaluation?
How does evaluation differ from monitoring performance? Here are some key features that will help us distinguish between them.
– Evaluation is a way of measuring progress toward established outcomes. It’s more in-depth than program monitoring in that a plan is created that often uses several performance measures together
– Evaluation makes an objective assessment of effectiveness based on a systematic collection of data
– It’s a time-limited effort used to understand the extent to which program outcomes have been achieved. It can also help to explore the unanticipated impacts of our intervention.
– Program monitoring and data collection are a part of evaluation.
– Evaluation can measure the process of program delivery as well as the outcomes
– Evaluation can be done at different times in a program lifespan including before, during or after a program has been delivered.
– Evaluation can be used to measure things other than programs including overall operational effectiveness, governance, or financial performance.
– Evaluation can be applied to assess a broader system across programs and/or delivery sites.
Why choose evaluation in the first place?
There are several reasons why we would choose evaluation as a useful tool in our nonprofit environment.
Firstly, it can help to build a case for support with donors or those wishing to fund our efforts. Evaluation helps us to gain insights into what’s working, what isn’t and how we can make improvements. Ultimately, evaluation findings will help us articulate the need for what we do and how that aligns with funder and donor priorities.
Evaluation helps us develop a deeper understanding of client needs and experiences. The better we understand our clients, the more responsive we can be to their needs and strengthen client relations. It can also help us identify gaps in who we are not serving so we can make plans to include them.
Lastly, evaluation helps to build an organization’s visibility and credibility. Conducting an evaluation signals responsible stewardship of the money, time and people invested in our work. If we find ways to include external stakeholders in our evaluation, this effect is extended to include the broader community.
When, then, do we choose to evaluate?
There are certain times when it’s useful and appropriate to choose to evaluate.
Evaluations are part of a management cycle and as such can be done at regular intervals. It can also be done when a program has launched, as it can help to show proof of concept and lay the groundwork for future planning and proposal development.
Lastly, evaluation can help us forge innovative new directions. If we evaluate a program or service that is performing less successfully that we hope, it can help us learn why and set us on a path to better meet the changing needs of our community.
How do we know when to call in an external evaluator?
While building our internal capacity in evaluation is important, there are times when an external evaluator is needed. Here are some tips that can help us assess our nonprofits evaluation needs.
Using a tool or two, we can do an initial assessment, with staff and board, to help us define our reasons for doing an evaluation. Try PREP’s evaluation readiness quiz to assess whether you’re prepared to start an evaluation. https://peelevaluates.ca/wp-content/uploads/2018/05/PREP_Download_Eval-Readiness-Quiz.pdf
When considering evaluation, it’s helpful to have conversations early on that help us identify what we want to learn from an evaluation, what decisions we want to make as a result of evaluation findings, and who the primary audience(s) for the evaluation will be. For some guidance on what to cover, try the Ontario Nonprofit Network’s Evaluation Discussion Guide: https://peelevaluates.ca/wp-content/uploads/2018/05/ONN-Evaluation-Discussion-Guide.pdf
Taking stock of our internal bench strength in evaluation is also an important step. Do we have existing skills in evaluation within our staff complement? In addition, are we able to allocate their time towards the planning and delivery of an evaluation?
We can delve into these questions and more through this resource that will help assess where our nonprofit falls on the continuum of mostly relying on external help to relying on internal evaluators. It also provides tips on engaging a professional evaluator: https://peelevaluates.ca/wp-content/uploads/2018/05/PREP_Download-SE-Checklist.pdf
Using Evaluation to Make A Difference
Rather than causing intimidation, the ‘aha moment’ experienced by our workshop participant motivated them to undertake a full-fledged evaluation. They have since shared with us that the experience has been a richly rewarding activity that engaged their clients and staff in addition to renewing their board’s connection to their organizational mission.
Here are some additional PREP resources that can help determine what kind of evaluation is right for your organization: