How many is enough? The quest for an acceptable survey response rate
September 16, 2009 at 3:31 pm 11 comments
How many people do I need to survey? That’s the most common question I get as a researcher. The answer, unfortunately, is not simple.
The goal of a survey is to give an accurate picture of how people feel about a topic and predict what they would do (e.g., buy, vote). In public opinion surveys, we often want to get an answer that is within a few percentage points of “truth.” A statistic called the standard error of the mean measures this wiggle room and is inversely related to the number of respondents. So from this perspective, the answer to “How many is enough” depends on how far off we’re willing to be.
Why response rate matters
In general, there are four sources of error in survey research and three of them relate to the people surveyed. (The fourth has to do with problems in the way the questions are designed.) Mistakes can be made in defining the population (e.g., people in special needs schools and institutions were not included in the original IQ test standardization). The sampling process may result in different demographics from the population (e.g., picking numbers at random from the White Pages eliminates the ~30% with an unlisted number). Or the responders may differ from the non-responders in some important way.
To get an accurate picture, we need to survey a representative sample of the population we are interested in. If the population of interest is small, we may send surveys to everyone. When we want to know what the general population thinks, we use random sampling or stratified random sampling, which sets quotas for the number of people sampled from categories of age, income, gender and/or other demographics. If the sampling process is sound and response rate is acceptably high, we can trust that the people who opt out do not differ in critical ways from the people who complete the survey. When the response rate is low, we may question the representativeness of the responders and validity of the results.
Measuring response rate
Although response rate is defined as the number of people we surveyed compared to the number of people we tried to survey, there is more than one way to measure it. The number of people surveyed may include surveys with some answers left blank or only those where every question was answered. The number we tried to survey may include only those known to fit our criterion for eligibility or also those on the mailing or phone list who are not eligible.
In actual fact, what some people report as response rate is actually their cooperation rate (i.e., the number of completed surveys compared to the number of people reached). The American Association for Public Opinion Research (AAPOR), an authority on all things survey-related, accepts several different formulas for response rate and asks that researchers disclose how they measured it in reports.
Survey mode makes a difference
The survey method has an impact on response rate. Generally, e-mail surveys have a lower response rate than mail surveys, even when access to the Internet is not an issue. For example, in a 2004 survey of university undergrads with e-mail access, about 21% responded to an e-mail survey while 31% responded to a mail survey (Kaplowitz et al., 2004 in Public Opinion Quarterly pp. 94-101). Face-to-face surveys achieve the highest response rates, with the best I’ve seen being a whopping 92%. Some studies report that telephone surveys have a higher response rate than mail surveys, while others report the reverse. Sending reminders boosts response rates. Oddly enough, studies have shown that sending a $2 incentive boosts both response rate and representativeness.
When I said that there is no simple answer to the question of how many is enough, this does not mean that people have been unwilling to go on record with a numerical answer. Here are some expert opinions as to what is considered good or adequate as a mail survey response rate:
- 25% – Dr. Norman Hertz when asked by the Supreme Court of Arizona
- 30% – R. Allen Reese, manager of the Graduate Research Institute of Hull U. in the United Kingdom
- 36% – H. W. Vanderleest (1996) response rate achieved after a reminder
- 38% – in Slovenia where surveys are uncommon
- 50% – Babbie (1990, 1998)
- 60% – Kiess & Bloomquist (1985) to avoid bias by the most happy/unhappy respondents only
- 60% – AAPOR study looking at minimum standards for publishability in key journals
- 70% – Don A. Dillman (1974, 2000)
- 75% – Bailey (1987) cited in Hager et al. (2003 in Nonprofit and Voluntary Sector Quarterly, pp. 252-267)
In addition, various studies described their response rate as “acceptable” at 10%, 54%, and 65%, while others on the American Psychological Association website reported caveats regarding non-responder differences for studies with 38.9%, 40% and 42% response rates.
It’s enough to make one’s head spin. And population surveys of employees or group members have different standards for acceptable response rates than the general population. In smaller populations such as these, it becomes difficult to compare subsets of the overall group, even with a high response rate, because the statistics lack the necessary power.
To make matters worse, there is good evidence that response rates for telephone surveys are declining. Reasons for this include increasing concerns about invasion of privacy and misuse of personal information; increased use of call display to filter calls; cynicism (especially among Gen Xers); reduced civic participation in general; and increased requests for survey participation—particularly in satisfaction surveys and program evaluations. (For example, my last car purchase resulted in three surveys from the dealership and Honda Canada.)
It’s about representativeness
Response rate is not the best way to judge the accuracy of survey results, but representativeness of respondents is. Not all demographic characteristics make a difference. Gary Langer of the ABC News Polling Unit described an instance in which repeated surveys gave the same pattern of answers, even though the respondents differed significantly in demographic characteristics. The main advantage of a high rate of response is in reducing the possibility of a non-representative sample.
Entry filed under: Management, Research. Tags: representativeness, Research, response rate, survey.
11 Comments Add your own
Leave a comment
Trackback this post | Subscribe to the comments via RSS Feed
Bright Ideas 
1.
Tina Dole | September 4, 2012 at 11:31 am
I am thinking about participating in paid surveys. I have a few friends that do them and make pretty good money. It seems like an easy, worthwhile job.
2.
kkbiersdorff | September 4, 2012 at 11:55 am
Hi Tina. I respond to surveys for frequent flier points and I can tell you that you will not “make pretty good money.” So if your comment here is just for advertising, you may want to withdraw it now. While most of the ones I respond to are legitimate and well-thought out, a few have glaring glitches, while others collect more information than they need to be sure you “qualify.”
These surveys use a “stratified random sampling process,” which means that they use the breakdown of population by gender, age, income and education (at the very least) to ensure that the sample characteristics match the population characteristics. So, if their sample already has enough responses from your gender, age group, etc., you will “not qualify” for the remainder of the survey and, as a result, get a minimal payout. On the other hand, you will have expended minimal time, and, as they say, time is money. These surveys have potential flaws. For instance, they are only completed by people who use the Internet and who are willing to do Internet market surveys. If the population whose behaviour you want to predict are Internet users (and Internet survey completers), then you are fine. If you are hoping to sell to a market that includes people not addicted to the Internet, then your sample will give you unhelpful information. They also eliminate people who work for PR and marketing companies and, sometimes, people in those industries who are independent. I understand the reasons, but it does limit their sample characteristics.
In short (and when am I ever), while stratified random sampling reduces the number of respondents needed, anything that skews the sample from the general population is going to have a negative impact on representativeness.
3.
Paula J. MacLean | February 11, 2014 at 6:26 pm
Kathleen, do you have a comment on adequate response rates for employers doing employee satisfaction surveys? The issue I encounter is: at what percentage can we reasonably assume that the results are representative of the majority of employees?
4.
kkbiersdorff | February 24, 2014 at 4:23 pm
Again, there is no easy answer, Paula. One question I would ask is how big and diverse is the organization. In a small organization–say 20 employees–you need a really high response rate because one person’s response makes a big difference to the total percent. For example, if 10 employees respond and 1 changes his/her mind, the result is a change of 10% in level of agreement or satisfaction. With 20 respondents, the single individual affects the overall level of agreement/satisfaction by 5%. Therefore, a small organization is better off not using survey technologies to gauge employee satisfaction. Focus groups (without supervisors) give better information and increase employee engagement at the same time (or can, if the organization demonstrates that it has listened to what they said).
With a larger organization, my answer would depend on its diversity and whether the leaders want to understand the differences between the different units or departments. Any time you break the large organization into smaller units, you run into the same issue of small numbers having a big impact on the percentages.
If you have a large, non-diverse organization, you do not need as high a response rate in order to have interpretable data. For example, if 100 employees respond out of 500 staff (a 20% response rate), I would be just as confident about the results as if 100 employees respond out of 1000 staff (a 10% response rate), assuming that both sets of respondents are representative of the organization’s employees. If, however, managers want to be able to compare results from different departments, then the 100 are divided into smaller chunks and my analyses start to lose power and each subset is less likely to be representative of its department or unit.
In general, the people who are most likely to take the time to respond to a satisfaction survey are those who are very happy and want you to know it, and those who are very unhappy and want you to know it. The ones in between may respond if you provide an unrelated incentive to do so, such as the opportunity to win one of your fabulously useful books, Paula. Valued incentives increase both response rate and representativeness.
From a statistics standpoint, a sample size (number of respondents) of 30 approximates a normal distribution. I am typically uncomfortable with fewer respondents unless the organization is small. Otherwise, the percentage that I am comfortable with decreases as the size of the organization increases. Ultimately, the only way to ensure representativeness is through an analysis of the demographics of the organization and your respondents. There, I would say that age, gender and role or unit in the organization would be the most important factors to measure. However, asking for job title or unit may make some people hesitate to respond for fear of losing anonymity. It’s a balancing act.
5.
Welcome to the National Social Norms Institute. – What is an acceptable survey response rate? | November 12, 2014 at 10:29 am
[…] is from a 2009 web paper by Kathy Biersdorff who is a business consultant in the Calgary […]
6.
kkbiersdorff | November 24, 2014 at 6:50 pm
An interesting and updated article for those wanting more information (but not definitive answers, of course) on this same topic.
7.
Patricia Adams | January 28, 2015 at 7:42 am
I think online surveys are a better way to gain more responses. I have published surveys on social media and have received good no.of responses
8.
kkbiersdorff | January 28, 2015 at 5:18 pm
The research data say that online surveys have the lowest response rates of any type of feedback gathering process. If you put an online survey out there on social media, you will (with luck) get a lot of responses, but the percentage of responses (that is number of responses divided by the number of people who had access to the survey times 100) will be low because the population using social media is quite high. I am assuming, of course, that the social media you are using is not a closed group but, for instance, all Twitter users. The important consideration is not how many responses you get per se, but how representative they are of the population or sub-population whose behaviour or attitudes you wish to predict. Now let me move from professional to rude and obnoxious. You started by saying “I think” which suggests that you are giving an opinion based on personal experience. That is fine for issues where data have not been collected extensively. But where sound data exist, it is possible to KNOW, at which point to think something contrary to the data is not smart. I apologize if I was reading more into you choice of words.
9.
Ronnie | December 29, 2015 at 8:13 am
Nice piece, I have learned much.
10.
junewilliams7 | June 9, 2017 at 12:28 pm
Thank you for this article; it is very helpful.
Question: I’d like to find the source of this figure: “25% – Dr. Norman Hertz when asked by the Supreme Court of Arizona”. Would you happen to recall where you found it? I tried Googling but didn’t find it, and couldn’t find an email address for Dr Hertz. Thank you.
11.
kkbiersdorff | June 9, 2017 at 1:19 pm
As with many online sources, this one seems to have disappeared from the Internet and is not in any document that I downloaded into my files. I can tell you that it did not come from a personal communication with Dr. Hertz and I do not have any references in my notes to having read Arizona Supreme Court case files, because I would have cited the case. Everything points to it having been a secondary source. If I had to guess where I would have run across it, I would guess in the work of Don Dillman, who I consider to be one of the best sources of sound information in this area. If you re-read the particular section of my little paper, you will see that the point of the bit about Dr. Hertz is that there is no consensus on the response rate required to be sound, but that various people are willing to go on record identifying a number. Therefore, response rate is not nearly as critical as being able to establish that your sample is REPRESENTATIVE of the population of interest.