How many is enough? The quest for an acceptable survey response rate

September 16, 2009 at 3:31 pm 4 comments

How many people do I need to survey? That’s the most common question I get as a researcher. The answer, unfortunately, is not simple.

The goal of a survey is to give an accurate picture of how people feel about a topic and predict what they would do (e.g., buy, vote). In public opinion surveys, we often want to get an answer that is within a few percentage points of “truth.” A statistic called the standard error of the mean measures this wiggle room and is inversely related to the number of respondents. So from this perspective, the answer to “How many is enough” depends on how far off we’re willing to be.

Why response rate matters

In general, there are four sources of error in survey research and three of them relate to the people surveyed. (The fourth has to do with problems in the way the questions are designed.) Mistakes can be made in defining the population (e.g., people in special needs schools and institutions were not included in the original IQ test standardization). The sampling process may result in different demographics from the population (e.g., picking numbers at random from the White Pages eliminates the ~30% with an unlisted number). Or the responders may differ from the non-responders in some important way.

To get an accurate picture, we need to survey a representative sample of the population we are interested in. If the population of interest is small, we may send surveys to everyone. When we want to know what the general population thinks, we use random sampling or stratified random sampling, which sets quotas for the number of people sampled from categories of age, income, gender and/or other demographics. If the sampling process is sound and response rate is acceptably high, we can trust that the people who opt out do not differ in critical ways from the people who complete the survey. When the response rate is low, we may question the representativeness of the responders and validity of the results.

Measuring response rate

Although response rate is defined as the number of people we surveyed compared to the number of people we tried to survey, there is more than one way to measure it. The number of people surveyed may include surveys with some answers left blank or only those where every question was answered. The number we tried to survey may include only those known to fit our criterion for eligibility or also those on the mailing or phone list who are not eligible.

In actual fact, what some people report as response rate is actually their cooperation rate (i.e., the number of completed surveys compared to the number of people reached). The American Association for Public Opinion Research (AAPOR), an authority on all things survey-related, accepts several different formulas for response rate and asks that researchers disclose how they measured it in reports.

Survey mode makes a difference

The survey method has an impact on response rate. Generally, e-mail surveys have a lower response rate than mail surveys, even when access to the Internet is not an issue. For example, in a 2004 survey of university undergrads with e-mail access, about 21% responded to an e-mail survey while 31% responded to a mail survey (Kaplowitz et al., 2004 in Public Opinion Quarterly pp. 94-101). Face-to-face surveys achieve the highest response rates, with the best I’ve seen being a whopping 92%. Some studies report that telephone surveys have a higher response rate than mail surveys, while others report the reverse. Sending reminders boosts response rates. Oddly enough, studies have shown that sending a $2 incentive boosts both response rate and representativeness.

When I said that there is no simple answer to the question of how many is enough, this does not mean that people have been unwilling to go on record with a numerical answer. Here are some expert opinions as to what is considered good or adequate as a mail survey response rate:

  • 25% – Dr. Norman Hertz when asked by the Supreme Court of Arizona
  • 30% – R. Allen Reese, manager of the Graduate Research Institute of Hull U. in the United Kingdom
  • 36% – H. W. Vanderleest (1996) response rate achieved after a reminder
  • 38% – in Slovenia where surveys are uncommon
  • 50% – Babbie (1990, 1998)
  • 60% – Kiess & Bloomquist (1985) to avoid bias by the most happy/unhappy respondents only
  • 60% – AAPOR study looking at minimum standards for publishability in key journals
  • 70% – Don A. Dillman (1974, 2000)
  • 75% – Bailey (1987) cited in Hager et al. (2003 in Nonprofit and Voluntary Sector Quarterly, pp. 252-267)

In addition, various studies described their response rate as “acceptable” at 10%, 54%, and 65%, while others on the American Psychological Association website reported caveats regarding non-responder differences for studies with 38.9%, 40% and 42% response rates.

It’s enough to make one’s head spin. And population surveys of employees or group members have different standards for acceptable response rates than the general population. In smaller populations such as these, it becomes difficult to compare subsets of the overall group, even with a high response rate, because the statistics lack the necessary power.

To make matters worse, there is good evidence that response rates for telephone surveys are declining. Reasons for this include increasing concerns about invasion of privacy and misuse of personal information; increased use of call display to filter calls; cynicism (especially among Gen Xers); reduced civic participation in general; and increased requests for survey participation—particularly in satisfaction surveys and program evaluations. (For example, my last car purchase resulted in three surveys from the dealership and Honda Canada.)

It’s about representativeness

Response rate is not the best way to judge the accuracy of survey results, but representativeness of respondents is. Not all demographic characteristics make a difference. Gary Langer of the ABC News Polling Unit described an instance in which repeated surveys gave the same pattern of answers, even though the respondents differed significantly in demographic characteristics. The main advantage of a high rate of response is in reducing the possibility of a non-representative sample.

Entry filed under: Management, Research. Tags: , , , .

Social media: Time waster or valuable business tool Finding the “Truth”: Using surveys to find out what people really think

4 Comments Add your own

  • 1. Tina Dole  |  September 4, 2012 at 11:31 am

    I am thinking about participating in paid surveys. I have a few friends that do them and make pretty good money. It seems like an easy, worthwhile job.

    Reply
    • 2. kkbiersdorff  |  September 4, 2012 at 11:55 am

      Hi Tina. I respond to surveys for frequent flier points and I can tell you that you will not “make pretty good money.” So if your comment here is just for advertising, you may want to withdraw it now. While most of the ones I respond to are legitimate and well-thought out, a few have glaring glitches, while others collect more information than they need to be sure you “qualify.”

      These surveys use a “stratified random sampling process,” which means that they use the breakdown of population by gender, age, income and education (at the very least) to ensure that the sample characteristics match the population characteristics. So, if their sample already has enough responses from your gender, age group, etc., you will “not qualify” for the remainder of the survey and, as a result, get a minimal payout. On the other hand, you will have expended minimal time, and, as they say, time is money. These surveys have potential flaws. For instance, they are only completed by people who use the Internet and who are willing to do Internet market surveys. If the population whose behaviour you want to predict are Internet users (and Internet survey completers), then you are fine. If you are hoping to sell to a market that includes people not addicted to the Internet, then your sample will give you unhelpful information. They also eliminate people who work for PR and marketing companies and, sometimes, people in those industries who are independent. I understand the reasons, but it does limit their sample characteristics.

      In short (and when am I ever), while stratified random sampling reduces the number of respondents needed, anything that skews the sample from the general population is going to have a negative impact on representativeness.

      Reply
  • 3. Paula J. MacLean  |  February 11, 2014 at 6:26 pm

    Kathleen, do you have a comment on adequate response rates for employers doing employee satisfaction surveys? The issue I encounter is: at what percentage can we reasonably assume that the results are representative of the majority of employees?

    Reply
    • 4. kkbiersdorff  |  February 24, 2014 at 4:23 pm

      Again, there is no easy answer, Paula. One question I would ask is how big and diverse is the organization. In a small organization–say 20 employees–you need a really high response rate because one person’s response makes a big difference to the total percent. For example, if 10 employees respond and 1 changes his/her mind, the result is a change of 10% in level of agreement or satisfaction. With 20 respondents, the single individual affects the overall level of agreement/satisfaction by 5%. Therefore, a small organization is better off not using survey technologies to gauge employee satisfaction. Focus groups (without supervisors) give better information and increase employee engagement at the same time (or can, if the organization demonstrates that it has listened to what they said).
      With a larger organization, my answer would depend on its diversity and whether the leaders want to understand the differences between the different units or departments. Any time you break the large organization into smaller units, you run into the same issue of small numbers having a big impact on the percentages.
      If you have a large, non-diverse organization, you do not need as high a response rate in order to have interpretable data. For example, if 100 employees respond out of 500 staff (a 20% response rate), I would be just as confident about the results as if 100 employees respond out of 1000 staff (a 10% response rate), assuming that both sets of respondents are representative of the organization’s employees. If, however, managers want to be able to compare results from different departments, then the 100 are divided into smaller chunks and my analyses start to lose power and each subset is less likely to be representative of its department or unit.
      In general, the people who are most likely to take the time to respond to a satisfaction survey are those who are very happy and want you to know it, and those who are very unhappy and want you to know it. The ones in between may respond if you provide an unrelated incentive to do so, such as the opportunity to win one of your fabulously useful books, Paula. Valued incentives increase both response rate and representativeness.
      From a statistics standpoint, a sample size (number of respondents) of 30 approximates a normal distribution. I am typically uncomfortable with fewer respondents unless the organization is small. Otherwise, the percentage that I am comfortable with decreases as the size of the organization increases. Ultimately, the only way to ensure representativeness is through an analysis of the demographics of the organization and your respondents. There, I would say that age, gender and role or unit in the organization would be the most important factors to measure. However, asking for job title or unit may make some people hesitate to respond for fear of losing anonymity. It’s a balancing act.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 8 other followers

Recent Posts


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: