You Still Need High Response Rates with Web-Based Surveys

As the student affairs profession continues to search for solutions to its growing need to assess services and activities, more professionals are turning to Web-based surveys as a quick means of gathering data. Unfortunately, many of the professionals who are employing Web-based survey technology do not understand the science of survey research. While a Web-based survey certainly can be an easy way to collect data, the real question is what do those data mean? Are the data you gather meaningful relative to the population you are trying to study?

The Alumni Association at my campus recently conducted a web-based survey where the methodology was fairly typical of many such surveys that I have seen. Thousands of alumni were invited to fill out a survey on the Web, and the survey produced over 400 responses (a reported 9% response rate). So, how should we interpret those 400+ responses?

While 400 respondents can be an adequate number, it depends upon how many potential respondents were invited to participate in the survey. If you have invited 5000 people to respond, then 400 responses are not adequate. Based on this example, if 90% of the respondents indicated that they were very satisfied with their student experience at UMass, would you conclude that our alumni in general were very satisfied with their experiences as students on campus? You might draw that conclusion, but based on these data, you stand a good chance of being wrong. The problem lies in what survey researchers refer to as non-response bias. Perhaps, only the folks with positive experiences responded to the survey. We do not know the opinions of the folks who did not respond, and because so many alumni chose not to respond, the data that were collected are virtually useless as a gauge of the opinions of the general population of alumni invited to participate in the survey. The only way that you can feel confident that survey data are representative of a population is to have a high response rate from the people you attempted to contact (either the entire population or a large enough random sample of that population). Generally, around 60% response rates are acceptable, although Dillman (1978, 2000) encourages closer to 80% while acknowledging that such high rates are very difficult to obtain. It also is important that the demographic data (e.g. age, sex, race, class year) of the sample closely matches the demographic data of the population that the sample is intended to represent.

So, how do you go about securing a high response rate from a Web-based survey? If your population is large (for purposes here, let’s say over 2000), you do not open up the survey to the entire population. One of my fears of Web-based data collection, as Dillman (2000) has said, is that “researchers” will design a Web-based instrument and solicit data from anyone willing to complete the survey. Administrators must understand the importance of sampling (see Sudman, 1976) in all data collection efforts, or else they run the risk of obtaining very low response rates and useless data.

Once you have selected an appropriate sample, you need to develop a protocol for improving the response rate. In other words, follow-up with the non-respondents until they become respondents. Sending several email reminders is a logical practice. In one of his examples, Dillman (2000) noted the importance using a telephone call reminder. Another consideration is a raffle along with telephone follow-ups (Lam, Malaney, & Oteri, 1990). Gary Hanson (personal communication, January 27, 2001) indicated that a key practice that has helped him obtain high response rates for Web-based surveys is sending a letter to each individual in the sample before using email solicitations.

Most importantly, we should not be encouraging individuals who have no formal training in survey research to be conducting any type of survey (Web-based or otherwise) without first doing considerable homework. The two books by Don Dillman that I have cited in this article are absolutely essential reading: Mail and Telephone Surveys: The Total Design Method (1978) and Mail and Internet Surveys: The Tailored Design Method (2000). These texts provide information that is crucial to obtaining high response rates for surveys. Virtually everyone in the survey business still relies on the methods detailed by Dillman over 20 years ago. Other authors also have provided helpful tips pertaining to Web-based surveys (e.g., Williams & Anderson, 1999; Wortman & Upcraft, 2001).

References

Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: John Wiley.

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method. New York: John Wiley.

Lam, J. A., Malaney, G. D., & Oteri, L. A. (1990). Strategies to increase student response rates to mail surveys. Journal of Marketing for Higher Education, 3(1), 89-105.

Sudman, S. (1976). Applied sampling. New York: Academic Press.

Williams, E. A., & Anderson, C. M. (1999). Applications of technology to assist student affairs researchers. In G. D. Malaney (Ed.), Student affairs research, evaluation, and assessment: Structure and practice in an era of change (New Directions for Student Services, No. 85, pp. 61-71). San Francisco: Jossey-Bass.

Wortman, T. I., & Upcraft, M. L. (2001). Web-based data collection. In J. H. Schuh & M. L. Upcraft, Assessment practice in student affairs: An applications manual (pp. 101-125). San Francisco: Jossey-Bass.