1. Section Introduction
Data is a key part of evaluation but for some, the word ‘data’ can be intimidating. This section helps to dispel this by explaining what data is and the different types of data that can be collected in evaluation. Next, we offer ways to develop indicators that will help determine what specific data will be needed to support our evaluation goals. What follows is a primer on data integrity- what it is and tips for including it in evaluation activities. Moving along the continuum of evaluation is a collection of resources outlining the steps and procedures that are a part of analyzing or making sense of the data once it’s gathered. We finish by talking about how to tell stories with our data through data visualization, so that we’re able to share our evaluation findings in a way that is accessible and engaging.
2. What is Data?
What is ‘data’ and where do I get it?
Evaluation always involves gathering data. Data is factual information, is systematically organized and used to help someone make a decision. To conduct a successful evaluation, we need to identify strategies for gathering appropriate data and evidence. The data collected should align with the evaluation objectives and should seek to answer the evaluation questions.
We can begin by looking at the data already being collected to see whether existing data can adequately answer the evaluation questions.
Existing data (secondary data) are data we already collect such as performance measures or data that are gathered by external sources, such as administrative data. Administrative data may include client records and Census data.
If existing data are not available to answer our particular evaluation questions, we’ll need to collect new data (primary data). Most people associate data with numbers: dollars, demographics, per cents or averages. Statistical data, usually called quantitative data, is part of most evaluations, but is by no means the only kind of information that is useful to nonprofit organizations.
3. Qualitative and Quantitative Data
What types of data should we be collecting when we evaluate? Is there one type of data that is considered better than another?
Data gathered for program evaluation can be qualitative or quantitative. Depending on our goals, one type may be better suited to meet our needs, however a mixed method of both quantitative and qualitative data can enrich our evaluation and provide a more comprehensive set of results.
The type of data we collect will influence the plan and approach we take. Depending on the evaluation questions being asked, there are several different types of data collection instruments that can be developed and administered to collect new data.
4. Linking Questions, Outcomes and Indicators
Evaluation questions are intended to broadly define what will be assessed. Evaluation questions can focus on:
- Planning and implementation (e.g., how well was the program was planned out and how well was the plan put into practice?)
- Attainment of program objectives (e.g., how well has the program met its stated objectives?)
- Program impact (e.g., what difference has the program made to its intended targets of change or community as a whole?)
To operationalize evaluation questions, they need to be linked to key program outputs and outcomes. From here, measures or indicators are developed for each one. An indicator is a factor used to measure or demonstrate change.
5. Data Integrity
What is data integrity and why does it matter?
Data integrity refers to the accuracy and consistency of data. When embarking on an evaluation, it’s important to have processes in place so that the data is captured in a way that preserves its reliability. Being explicit about how data will be managed and checked during an evaluation increases trust in our evaluation results for our own benefit and for that of our stakeholders and community.
6. Making Sense of Your Data
In this section, we’ll learn about the most common quantitative analysis procedures that are used in small program evaluation. We’ve included a template to help plan for data analysis. Following this, we take a deeper dive into different kinds of measurement and learn about a variety of data analysis procedures from basic to advanced. There is much more that can be learned about data analysis, so a resource list specifically about this topic is also included here.
7. Data Visualization
Using data visualizations, such as charts and infographics, help to make our evaluation results more compelling. It enables us to tell a story with our data that is more accessible than a written report. We’ve curated a list of websites and resources that facilitate getting started with data visualization and provide some tips for visual design.
- Pell Institute – Evaluation 101 Worksheet
- Alberta Health Services – Eval Resources Plan Framework
- Excellence for Child and Youth – Program Evaluation Toolkit
- Western Michigan University – Indicator Checklist
- Academia – Three Steps for Keeping Audiences Riveted When Talking About Data
Next: Sharing Evaluation