1. Section Introduction
There are many reasons to learn more about evaluation. This section will be helpful for anyone who is:
- Planning an evaluation for the first time
- Wants to enrich their understanding of evaluation in order to incorporate it more fully into grant proposals
- Aims to build their organization’s internal capacity in evaluation and intends to share this information with their colleagues, boards or staff teams.
In this section, we’ll dispel some common myths about evaluation, and describe the ways in which evaluation can strengthen our nonprofit organizations.
Then, we’ll introduce a number of questions that nonprofits can consider before planning for evaluation. We wrap up this section with an orientation to the types of program evaluation and how they can be used.
2. Program Evaluation – what it is and what it isn’t
Program evaluation is often seen as a “necessary evil.” Something that funders require, but something we would gladly dispense with otherwise. After all, doesn’t the fact that we run our programs day to day show that we already know everything we need to know about them? Isn’t evaluation really just an obsession with data?
Let’s face it – we don’t always know what we don’t know. In our programs that perform well, it’s likely we found an approach that was suited to the existing needs at the time. How do we know if what we’re delivering is really having the effect we intended? Also, how do we adapt and hone our operations so we continue to deliver high quality programs and services into the future?
This is where evaluation comes in. A program evaluation, if done well will help determine the effectiveness of our programs and generate recommendations for improvement that will make them even more successful. This is particularly important when program funding is tied to results or outcomes. Evaluation is the way to assess progress against outcomes that is trusted and verifiable. It requires planning, but it’s worth it.
Sometimes we see program evaluation and performance monitoring as the same thing. While both can measure the extent that outcomes have been achieved, program evaluation is a more in-depth process that can use several performance measures together in the evaluation framework. Program evaluation is usually a time-limited effort that’s used to understand why or why not results have been achieved and to what extent the results can be attributed to the program. Evaluation studies may even seek to explore the unanticipated effects of a program or intervention.
We know there are many ideas about evaluation that make it seem difficult and out-of-reach for most nonprofits on modest budgets. Download our ‘common myths’ prepsheet to discover some perceived barriers to evaluation and how they can be addressed.
3. Benefits of Program Evaluation
There is tremendous pressure on nonprofits to prove their effectiveness and to demonstrate that they are achieving what they’ve set out to do. Because of this, accountability is a main reason for evaluation. Accountability to funders, to program participants, boards, staff and the public. Evaluation also provides a unique vantage point from which we can view what we do and how well we do it.
Evaluation can help us:
- Develop a deeper understanding of client needs and expectations
- Demonstrate program outcomes to funders and other stakeholders
- Substantiate a request for increased funding by providing evidence of effectiveness
- Drive improvements based on data collected about program performance
- Weigh program costs and resource requirements against client needs and program outcomes
- Document program performance that supports internal decision-making about changes to the program or staffing needs
- Gain insights used for strategic planning
- Help build our organization’s visibility and credibility
4. Standards of Evaluation
The Joint Committee on Standards for Educational Evaluation has published a set of standards for program evaluation. The Program Evaluation Standards include thirty statements in five categories: utility, feasibility, propriety, accuracy, and evaluation accountability. Knowing these standards and applying them during and evaluation activity can give an evaluation credibility with stakeholders and funders.
5. Getting Started
So, what exactly is Program Evaluation and what do I need to get started?
Program Evaluation can seem like a complex activity when we’re just starting out. That’s why planning for an evaluation is essential. We can begin by discussing our organization’s answers to these basic questions:
- What decisions do we want to be able to make as a result of our evaluation?
- Who are primary audiences for the results?
- What kinds of information do we need?
- When is this information needed?
- Where will we get the information and how?
- What resources are available to get the information, analyze it and report it?
- How will we report that information in a useful fashion?
Assessing our organization’s readiness to evaluate is an important first step. The Basic Ingredients Guide explains what needs to be in place before we decide if it is the right time for evaluation.
ONN’s Five Important Discussion Questions is a guide to having a conversation about evaluation with our board or other stakeholder groups. Finally, the Rainbow Framework is a guide for planning an evaluation through a series of questions and steps.
6. Exploring Types of Program Evaluation
The type of evaluation we decide to use depends on what we want to learn about our program. This decision builds off the previous step of building agreement on our organizations “why” of evaluation. Namely, what do we need to know that will help us make program decisions and how can we accurately collect and understand information that will help us gain this knowledge.
There are three main approaches to consider: formative, summative and developmental evaluations.
Formative Evaluation takes place before or during a project’s implementation with the aim of improving the project’s design and performance.
Formative evaluation helps us understand why a program works or doesn’t, and whether it’s being implemented as planned. This type of evaluation also explores what other factors (internal and external) are influencing the program. Because formative evaluation examines programs as they occur, it can provide feedback that allows us to make timely adjustments to improve program delivery.
Summative Evaluation, on the other hand, looks at the impact of a program on that program’s target group and examines program outcomes to determine the overall effectiveness of the program.
Summative evaluation is outcome-focused more than process-focused. Summative evaluation provides answers to questions such as:
- Were your program objectives met?
- Were there any unanticipated outcomes (positive or negative)?
- Will you need to improve and modify the structure of the program?
- What is the impact of the program?
- What resources will you need to address the program’s weaknesses?
Developmental Evaluation uses both Formative and Summative approaches. This goal of this evaluation type is to apply processes that will generate almost real-time feedback that’s used specifically to fuel development. This approach and the ways it’s best used, are explored in further depth in the next section.
Learn more about formative and summative evaluation by downloading these information sheets. Then download the checklist for some helpful questions to determine the type that will your organization’s needs.
7. Developmental Evaluation-is it right for your initiative?
What is Developmental Evaluation?
Development Evaluation (DE) is a process of evaluation that generates close-to real time feedback that’s used to foster development. DE is often used by social innovators to develop change initiatives in complex environments. In the social sector, it plays a similar role to that of research and development in the private sector. DE facilitates a continuous feedback loop that allows for concepts to be framed, delivered and tracked. Issues are identified quickly so that the approach can be adjusted and sent back through these series of steps. A DE approach can be employed for different kinds of innovations including new projects or programs, major organizational changes, policy reforms or system innovations (Patton, 2010). This method of evaluation has an internal focus where the evaluator works closely within a team to integrate processes of gathering and interpreting data that support the feedback loop.
Developmental Evaluation is also called: Real-time evaluation, Emergent evaluation, Action evaluation or Adaptive evaluation.
DE is best suited for organizations in which:
- innovation is identified as a core value;
- there is an iterative loop of option generation, testing and selection;
- board and staff are in agreement about innovation and willing to take risks;
- there is a high degree of uncertainty about the path forward;
- there are resources available for ongoing exploration; and
- the organization has a culture suited to exploration and enquiry
The following resources are adapted from J W McConnell Foundation:
8. Developing an evaluation mindset
Weaving evaluation activities into nonprofit operations involves developing an evaluation mindset. Part of that journey involves addressing any misunderstandings we may have about what evaluation may or may not mean for our organization’s future. We also need to determine how our evaluation activities will connect with our program goals or objectives and consider what kinds of stories we want to tell with the data we collect. In this pre-evaluation phase, this “Evaluation 101 worksheet” can help provide a foundation for evaluation thinking as it applies specifically to our nonprofit’s needs.
9. References for this section
- Evaluation: What it is and Why do it
- Types of Evaluation
- Evaluation Myths and Benefits
- Guide to Evaluation Outcomes
- Program Evaluation Guide
- Evaluation Toolbox
- Managing Program Evaluation Worksheet
Next: Planning for Evaluation