Using Surveys to Improve Impact Measurement

There are many tools we can use in the nonprofit sector to evaluate and collect data. Finding the most effective one for the task at hand, however, can be a tricky business.

Surveys are just one of these tools and may be something we’ve used over and over again in our day to day work. Surveys can be an efficient way of collecting information from a substantial number of people in order to answer evaluation research questions. For members of the nonprofit sector especially, surveys can help us gain valuable feedback and insights from donors and clients in order to better provide services for our communities.  

But is a survey the right method to use in our evaluation? The use of surveys and their role in evaluation was discussed at large last week, during a PREP evaluation workshop led by credentialed evaluator and founder of Social Impact Squared, Paul Bakker. Bakker enlightened an array of professionals in the non-profit sector about survey methods for outcome measurement.

How do we get the best and most accurate impact estimates, while considering all the issues that can potentially bias the results and survey participants? Bakker looked at the issues that makes the information inaccurate and shows us how to create better surveys. As we learned during this four-hour workshop, there are considerations that must be made when deciding if surveys will be the best way to improve impact measurement.  Here are just a few of the points discussed at length:

Let’s talk about accuracy…..

As useful as surveys are to collect information, they come with their own unique set of challenges when it comes to accuracy. Certain conditions have to be set in place in order for surveys to provide valid and trustworthy data. The group of people surveyed, for instance, has to be representative of the population we serve. Accuracy is also greatly impacted by sample size. The sample size has to be reasonably large, as the accuracy and precision of survey estimates increase the more people who are surveyed. With smaller sample sizes, Bakker observed, the precision goes down.

We also learned about ways to up our survey game by using pre and post surveys. Comparing outcomes across time, in this way, helps to increase the precision of our data.

We are more positive than we realize

Humans are positive-minded? Hard to believe, right? With all the negativity circling in our society, we actually tend to be too positive when it comes to surveys. As humans, we want to be “nice” and aren’t always truthful when it comes to giving opinions about how we feel an event or situation went. This makes it hard to tell how accurate the survey results truly are. This must be kept in mind when using surveys to evaluate data.

Direct observation vs. surveys

What’s more effective, handing out a survey or directly observing someone engaged in an activity? The main difference with direct observation is it’s often more accurate and precise than surveys, says Bakker. “With surveys, the limitation is you’re getting what people remember.” Bakker uses the example of asking how much TV someone watched in the span of a month. You’re relying on that person to accurately recall exactly how much television was viewed. It’s really an impossible task for even the sharpest mind. Compare this to a Nielsen box for TV ratings, which is very accurate and not based on memory.   

Direct observation may be a very precise and accurate way of measuring data, but the challenge is it’s more costly and not always practical. It’s simply not practical to follow a group or individual’s activities for a length of time.

Are you asking the right questions?

We have to ask the right questions to capture the right concept when using a survey as an accurate tool to measure data. For example, Bakker says, if you ask whether someone is good at school, is the responder answering the question based on confidence, or ability? An overly confident respondent could answer yes, while, his/her academic history tells a completely different story. It’s important to keep this in mind when thinking of using a survey.

Evaluator Paul Bakker offered great insight into the realm of survey methods and some factors we should consider when choosing to use a survey to measure our impact. For more information on Paul’s work, visit his website: https://socialimpactsquared.com

 

We would love to hear your thoughts on this. Email us at hello@peelevaluates.ca!

Follow Our Blog

Do you want to receive evaluation insights from PLC and our partners in your inbox? Enter your email address to follow our blog!