How many is enough? The quest for an acceptable survey response rate
How many people do I need to survey? That’s the most common question I get as a researcher. The answer, unfortunately, is not simple.
The goal of a survey is to give an accurate picture of how people feel about a topic and predict what they would do (e.g., buy, vote). In public opinion surveys, we often want to get an answer that is within a few percentage points of “truth.” A statistic called the standard error of the mean measures this wiggle room and is inversely related to the number of respondents. So from this perspective, the answer to “How many is enough” depends on how far off we’re willing to be.
Why response rate matters
In general, there are four sources of error in survey research and three of them relate to the people surveyed. (The fourth has to do with problems in the way the questions are designed.) Mistakes can be made in defining the population (e.g., people in special needs schools and institutions were not included in the original IQ test standardization). The sampling process may result in different demographics from the population (e.g., picking numbers at random from the White Pages eliminates the ~30% with an unlisted number). Or the responders may differ from the non-responders in some important way.
To get an accurate picture, we need to survey a representative sample of the population we are interested in. If the population of interest is small, we may send surveys to everyone. When we want to know what the general population thinks, we use random sampling or stratified random sampling, which sets quotas for the number of people sampled from categories of age, income, gender and/or other demographics. If the sampling process is sound and response rate is acceptably high, we can trust that the people who opt out do not differ in critical ways from the people who complete the survey. When the response rate is low, we may question the representativeness of the responders and validity of the results.
Measuring response rate
Although response rate is defined as the number of people we surveyed compared to the number of people we tried to survey, there is more than one way to measure it. The number of people surveyed may include surveys with some answers left blank or only those where every question was answered. The number we tried to survey may include only those known to fit our criterion for eligibility or also those on the mailing or phone list who are not eligible.
In actual fact, what some people report as response rate is actually their cooperation rate (i.e., the number of completed surveys compared to the number of people reached). The American Association for Public Opinion Research (AAPOR), an authority on all things survey-related, accepts several different formulas for response rate and asks that researchers disclose how they measured it in reports.
Survey mode makes a difference
The survey method has an impact on response rate. Generally, e-mail surveys have a lower response rate than mail surveys, even when access to the Internet is not an issue. For example, in a 2004 survey of university undergrads with e-mail access, about 21% responded to an e-mail survey while 31% responded to a mail survey (Kaplowitz et al., 2004 in Public Opinion Quarterly pp. 94-101). Face-to-face surveys achieve the highest response rates, with the best I’ve seen being a whopping 92%. Some studies report that telephone surveys have a higher response rate than mail surveys, while others report the reverse. Sending reminders boosts response rates. Oddly enough, studies have shown that sending a $2 incentive boosts both response rate and representativeness.
When I said that there is no simple answer to the question of how many is enough, this does not mean that people have been unwilling to go on record with a numerical answer. Here are some expert opinions as to what is considered good or adequate as a mail survey response rate:
- 25% – Dr. Norman Hertz when asked by the Supreme Court of Arizona
- 30% – R. Allen Reese, manager of the Graduate Research Institute of Hull U. in the United Kingdom
- 36% – H. W. Vanderleest (1996) response rate achieved after a reminder
- 38% – in Slovenia where surveys are uncommon
- 50% – Babbie (1990, 1998)
- 60% – Kiess & Bloomquist (1985) to avoid bias by the most happy/unhappy respondents only
- 60% – AAPOR study looking at minimum standards for publishability in key journals
- 70% – Don A. Dillman (1974, 2000)
- 75% – Bailey (1987) cited in Hager et al. (2003 in Nonprofit and Voluntary Sector Quarterly, pp. 252-267)
In addition, various studies described their response rate as “acceptable” at 10%, 54%, and 65%, while others on the American Psychological Association website reported caveats regarding non-responder differences for studies with 38.9%, 40% and 42% response rates.
It’s enough to make one’s head spin. And population surveys of employees or group members have different standards for acceptable response rates than the general population. In smaller populations such as these, it becomes difficult to compare subsets of the overall group, even with a high response rate, because the statistics lack the necessary power.
To make matters worse, there is good evidence that response rates for telephone surveys are declining. Reasons for this include increasing concerns about invasion of privacy and misuse of personal information; increased use of call display to filter calls; cynicism (especially among Gen Xers); reduced civic participation in general; and increased requests for survey participation—particularly in satisfaction surveys and program evaluations. (For example, my last car purchase resulted in three surveys from the dealership and Honda Canada.)
It’s about representativeness
Response rate is not the best way to judge the accuracy of survey results, but representativeness of respondents is. Not all demographic characteristics make a difference. Gary Langer of the ABC News Polling Unit described an instance in which repeated surveys gave the same pattern of answers, even though the respondents differed significantly in demographic characteristics. The main advantage of a high rate of response is in reducing the possibility of a non-representative sample.