1. Section Introduction
Planning is where the action with evaluation begins! Up first is a quiz that will help us assess our state of readiness to evaluate. This step is critical to understanding what evaluation capacities exist within our nonprofits and to what extent a professional evaluator may need to be engaged.
The second section defines summative evaluation and offers a tool the help us assess our need for external evaluation help. What follows are templates and tools to create an evaluation plan, establish an evaluation framework and identify practical and useful objectives for our evaluation. Documenting the steps we’ll take when collecting data are an important part of the evaluation planning process, so we’ve included some tips to get our documentation started. To finish off, we’ll talk about stories – how they help in evaluation and some guidance on how to collect them.
2. Evaluation Readiness
Where is our organization on the evaluation continuum?
Assessing our organization’s readiness to evaluate is an important first step. The Basic Ingredients Guide explains what needs to be in place before we decide if it is the right time for evaluation.
ONN’s Five Important Discussion Questions is a guide to having a conversation about evaluation with our board or other stakeholder groups. Finally, the Rainbow Framework is a guide for planning an evaluation through a series of questions and steps.
Before starting a program evaluation, it is good to take a look at the organizational strengths already in place to effectively support an evaluation. It’s also helpful to look at areas where learning and growth that can be fostered.
The questionnaire below helps organizations understand where they are on the evaluation continuum. The questions are divided into three important components:
- Organizational Readiness: Examines the commitment and culture at the organizational level to undertake evaluation.
- Program Readiness: Compares the design of the program to the actual program in operation
- Evaluation Readiness: Examines the history of and focus on evaluation
3. Exploring Types of Evaluation
The type of evaluation we decide to use depends on what we want to learn about our program. This decision builds off the previous step of building agreement on our organizations “why” of evaluation. Namely, what do we need to know that will help us make program decisions and how can we accurately collect and understand information that will help us gain this knowledge.
There are three main approaches to consider: formative, summative and developmental evaluations.
Formative Evaluation takes place before or during a project’s implementation with the aim of improving the project’s design and performance.
Formative evaluation helps us understand why a program works or doesn’t, and whether it’s being implemented as planned. This type of evaluation also explores what other factors (internal and external) are influencing the program. Because formative evaluation examines programs as they occur, it can provide feedback that allows us to make timely adjustments to improve program delivery.
Summative Evaluation, on the other hand, looks at the impact of a program on that program’s target group and examines program outcomes to determine the overall effectiveness of the program.
Summative evaluation is outcome-focused more than process-focused. Summative evaluation provides answers to questions such as:
- Were your program objectives met?
- Were there any unanticipated outcomes (positive or negative)?
- Will you need to improve and modify the structure of the program?
- What is the impact of the program?
- What resources will you need to address the program’s weaknesses?
Although it can take place during project implementation, it’s most often undertaken once a project is well underway. Unlike developmental evaluation, summative evaluation is outcome-focused more than process focused. It’s all about measuring and analyzing our impact through both the quantity and quality of our activities.
Developmental Evaluation uses both Formative and Summative approaches. This goal of this evaluation type is to apply processes that will generate almost real-time feedback that’s used specifically to fuel development. This approach and the ways it’s best used, are explored in further depth in the next section.
What is Developmental Evaluation?
Development Evaluation (DE) is a process of evaluation that generates close-to real time feedback that’s used to foster development. DE is often used by social innovators to develop change initiatives in complex environments. In the social sector, it plays a similar role to that of research and development in the private sector. DE facilitates a continuous feedback loop that allows for concepts to be framed, delivered and tracked. Issues are identified quickly so that the approach can be adjusted and sent back through these series of steps. A DE approach can be employed for different kinds of innovations including new projects or programs, major organizational changes, policy reforms or system innovations (Patton, 2010). This method of evaluation has an internal focus where the evaluator works closely within a team to integrate processes of gathering and interpreting data that support the feedback loop.
Developmental Evaluation is also called: Real-time evaluation, Emergent evaluation, Action evaluation or Adaptive evaluation.
DE is best suited for organizations in which:
• innovation is identified as a core value;
• there is an iterative loop of option generation, testing and selection;
• board and staff are in agreement about innovation and willing to take risks;
• there is a high degree of uncertainty about the path forward;
• there are resources available for ongoing exploration; and
• the organization has a culture suited to exploration and enquiry
Learn more about formative and summative evaluation by downloading these information sheets. Then download the checklist for some helpful questions to determine the type that will your organization’s needs.
The following resources are adapted from J W McConnell Foundation: https://mcconnellfoundation.ca/wp-content/uploads/2017/07/A-Developmental-Evaluation-Primer-EN.pdf
4. Exploring Frameworks
Frameworks are like roadmaps. They provide an overview of the work and are a vital link between operational goals and evaluation processes and activities. An evaluation framework is a tool used to organize and link evaluation questions, to program outcomes or outputs, indicators, data sources, and data collection methods.
5. Developing Smart Objectives
Naming the desired results of program activities by way of goals and objectives provides a context for evaluation and helps internal and external audiences understand the impact we want to make with our work. Ensuring those goals are measurable is very important. This section contains tools to help craft objectives that are SMART – Specific, Measurable, Actionable, Realistic and Timely.
6. Creating the Plan
Writing an evaluation plan is the final step to planning an evaluation. It’s a written document that describes the overall approach or strategy that will be used to guide the evaluation. It includes information on:
- Why the evaluation is being conducted
- What will be done
- Who will do it
- When will it be done
- How evaluation findings will likely be used
The plan should include a concise description of:
- The program and its goals
- Resources and scope of the evaluation
- Evaluation objectives and questions
- Outputs, outcomes and measures
- Data sources and data collection methods
- Ethical considerations
- Data analysis strategy
- Timelines and anticipated reporting dates
- Roles and responsibilities
- Strategy for disseminating results and developing recommendations
Below is an evaluation plan template that can serve as a guide to developing a robust program evaluation.
7. Documenting The Steps
Managing tasks associated with data collection is an important part of ensuring a successful evaluation. Developing a manual that documents the procedures for data collection can be useful for building internal capacity for ongoing evaluation. It also serves as a resource for staff involved in evaluation later in the process.
A data collection manual can include the following:
- An evaluation framework
- A detailed, step-by-step procedure
- A flowchart
- A copy of every measure being used in the evaluation
- Scoring manuals for each measure
- A copy of every consent form
- Participant tracking sheets
- Scripts that might be helpful for intake and administrative staff
- Location (electronic and hard copies) of all necessary materials
- A list of useful contacts
- Order forms for measures
- Instructions to complete the analysis (including syntax)
- Roles and responsibilities
- Better Evaluation
- W.K. Kellog Foundation: Step-by-Step Guide to Evaluation
- Evaluation 101 Worksheet
- Excellence for Child and Youth – Program Evaluation Toolkit
- Excellence for Child and Youth – Evaluation Framework Template Example
- Alberta Health Services – Evaluation Resources Plan Framework
- CDC – Smart Objectives
- Free Management Library – Program Evaluation Guide
- The Pell Institute – Quantitive and Qualitive Data
- Outcomes-Based Guide to Evaluation
- Planning Useful Evaluation Systems
Next: Understanding Data