1. Section Introduction
Planning is where the action with evaluation begins! Up first is a quiz that will help us assess our state of readiness to evaluate. This step is critical to understanding what evaluation capacities exist within our nonprofits and to what extent a professional evaluator may need to be engaged.
The second section defines summative evaluation and offers a tool the help us assess our need for external evaluation help. What follows are templates and tools to create an evaluation plan, establish an evaluation framework and identify practical and useful objectives for our evaluation. Documenting the steps we’ll take when collecting data are an important part of the evaluation planning process, so we’ve included some tips to get our documentation started. To finish off, we’ll talk about stories – how they help in evaluation and some guidance on how to collect them.
2. Evaluation Readiness
Where is our organization on the evaluation continuum?
Before starting a program evaluation, it is good to take a look at the organizational strengths already in place to effectively support an evaluation. It’s also helpful to look at areas where learning and growth that can be fostered.
The questionnaire below helps organizations understand where they are on the evaluation continuum. The questions are divided into three important components:
- Organizational Readiness: Examines the commitment and culture at the organizational level to undertake evaluation.
- Program Readiness: Compares the design of the program to the actual program in operation
- Evaluation Readiness: Examines the history of and focus on evaluation
3. Summative Evaluation
Summative evaluation is usually the method we’re referring to when we talk about program evaluation. Essentially, the aim of conducting a summative evaluation is to find out what’s been achieved through the work we do.
Although it can take place during project implementation, it’s most often undertaken once a project is well underway. Unlike developmental evaluation, summative evaluation is outcome-focused more than process focused. It’s all about measuring and analyzing our impact through both the quantity and quality of our activities.
4. Creating the Plan
Writing an evaluation plan is the final step to planning an evaluation. It’s a written document that describes the overall approach or strategy that will be used to guide the evaluation. It includes information on:
- Why the evaluation is being conducted
- What will be done
- Who will do it
- When will it be done
- How evaluation findings will likely be used
The plan should include a concise description of:
- The program and its goals
- Resources and scope of the evaluation
- Evaluation objectives and questions
- Outputs, outcomes and measures
- Data sources and data collection methods
- Ethical considerations
- Data analysis strategy
- Timelines and anticipated reporting dates
- Roles and responsibilities
- Strategy for disseminating results and developing recommendations
Below is an evaluation plan template that can serve as a guide to developing a robust program evaluation.
5. Exploring Frameworks
Frameworks are like roadmaps. They provide an overview of the work and are a vital link between operational goals and evaluation processes and activities. An evaluation framework is a tool used to organize and link evaluation questions, to program outcomes or outputs, indicators, data sources, and data collection methods.
6. Developing Smart Objectives
Naming the desired results of program activities by way of goals and objectives provides a context for evaluation and helps internal and external audiences understand the impact we want to make with our work. Ensuring those goals are measurable is very important. This section contains tools to help craft objectives that are SMART – Specific, Measurable, Actionable, Realistic and Timely.
7. Documenting The Steps
Managing tasks associated with data collection is an important part of ensuring a successful evaluation. Developing a manual that documents the procedures for data collection can be useful for building internal capacity for ongoing evaluation. It also serves as a resource for staff involved in evaluation later in the process.
A data collection manual can include the following:
- An evaluation framework
- A detailed, step-by-step procedure
- A flowchart
- A copy of every measure being used in the evaluation
- Scoring manuals for each measure
- A copy of every consent form
- Participant tracking sheets
- Scripts that might be helpful for intake and administrative staff
- Location (electronic and hard copies) of all necessary materials
- A list of useful contacts
- Order forms for measures
- Instructions to complete the analysis (including syntax)
- Roles and responsibilities
8. Finding Stories
Data + Story = Evidence
Finding and sharing stories of how our work is affecting the people we serve is an important part of evaluation. It showcases the human and experiential element of our work. Stories are highly relatable and memorable. Those who read our evaluation reports are more likely to remember a story we share than the numbers in a pie chart or graph. It’s just the way we’re wired!
In order for stories to be effective in evaluation, however, we need to think ahead about why we want to collect them and what we plan to do with them. Do we want to share evidence of our impact? Provide validation of a change that we are advocating for? Get an honest appraisal of what works and what doesn’t in our program?
Once we’re clear on our reasons to collect stories, we can plan for how and where we will gather them. Focus groups and stakeholder interviews can be great places to collect stories during the evaluation process.
Beyond stories we can collect from our staff, volunteers, board members and clients, we can also become attuned to the stories that naturally arise from our data and analysis during evaluation.
Check out the resources that follow to learn about different ways to plan for and utilize storytelling in our evaluation.
- Excellence for Child and Youth – Program Evaluation Toolkit
- Excellence for Child and Youth – Evaluation Framework Template Example
- Alberta Health Services – Evaluation Resources Plan Framework
- CDC – Smart Objectives
- Free Management Library – Program Evaluation Guide
- The Pell Institute – Quantitive and Qualitive Data
- Western Michigan University – Evaluation Framework Checklist
- The Goodman Centre – Free Range 2018
Next: Understanding Data