Join our mailing list!
©2007
StudentAffairs.com
All Rights Reserved
|
You
Still Need High Response Rates with Web-Based Surveys
Gary
D. Malaney
Director/Associate Professor
SARIS, 229 Whitmore
University of Massachusetts
Posted
February 15,
2002 Student
Affairs Online, 3 (Winter)
As the student affairs profession continues to search for
solutions to its growing need to assess services and activities, more
professionals are turning to Web-based surveys as a quick means of
gathering data. Unfortunately, many of the professionals who are
employing Web-based survey technology do not understand the science
of survey research. While a Web-based survey certainly can be an easy
way to collect data, the real question is what do those data mean?
Are the data you gather meaningful relative to the population you are
trying to study?
The Alumni Association at my campus recently conducted a web-based
survey where the methodology was fairly typical of many such surveys
that I have seen. Thousands of alumni were invited to fill out a
survey on the Web, and the survey produced over 400 responses (a
reported 9% response rate). So, how should we interpret those 400+
responses?
While 400 respondents can be an adequate number, it depends upon
how many potential respondents were invited to participate in the
survey. If you have invited 5000 people to respond, then 400
responses are not adequate. Based on this example, if 90% of the
respondents indicated that they were very satisfied with their
student experience at UMass, would you conclude that our alumni in
general were very satisfied with their experiences as students on
campus? You might draw that conclusion, but based on these data, you
stand a good chance of being wrong. The problem lies in what survey
researchers refer to as non-response bias. Perhaps, only the folks
with positive experiences responded to the survey. We do not know the
opinions of the folks who did not respond, and because so many alumni
chose not to respond, the data that were collected are virtually
useless as a gauge of the opinions of the general population of
alumni invited to participate in the survey. The only way that you
can feel confident that survey data are representative of a
population is to have a high response rate from the people you
attempted to contact (either the entire population or a
large enough random sample of that population). Generally, around 60%
response rates are acceptable, although Dillman (1978,
2000) encourages closer to 80% while
acknowledging that such high rates are very difficult to obtain. It
also is important that the demographic data (e.g. age, sex, race,
class year) of the sample closely matches the demographic data of the
population that the sample is intended to represent.
So, how do you go about securing a high response rate from a
Web-based survey? If your population is large (for purposes here,
let’s say over 2000), you do not open up the survey to the
entire population. One of my fears of Web-based data collection, as
Dillman (2000) has said, is that
“researchers” will design a Web-based instrument and
solicit data from anyone willing to complete the survey.
Administrators must understand the importance of sampling (see
Sudman, 1976) in all data collection efforts,
or else they run the risk of obtaining very low response rates and
useless data.
Once you have selected an appropriate sample, you need to develop
a protocol for improving the response rate. In other words, follow-up
with the non-respondents until they become respondents. Sending
several email reminders is a logical practice. In one of his
examples, Dillman (2000) noted the importance
using a telephone call reminder. Another consideration is a raffle
along with telephone follow-ups (Lam, Malaney, &
Oteri, 1990). Gary Hanson (personal communication, January 27,
2001) indicated that a key practice that has helped him obtain high
response rates for Web-based surveys is sending a letter to each
individual in the sample before using email solicitations.
Most importantly, we should not be encouraging individuals who
have no formal training in survey research to be conducting any type
of survey (Web-based or otherwise) without first doing considerable
homework. The two books by Don Dillman that I have cited in this
article are absolutely essential reading: Mail and Telephone
Surveys: The Total Design Method (1978)
and Mail and Internet Surveys: The Tailored Design Method
(2000). These texts provide information that
is crucial to obtaining high response rates for surveys. Virtually
everyone in the survey business still relies on the methods detailed
by Dillman over 20 years ago. Other authors also have provided
helpful tips pertaining to Web-based surveys (e.g., Williams
& Anderson, 1999; Wortman & Upcraft,
2001).
References
Dillman, D. A. (1978). Mail and
telephone surveys: The total design method. New York: John
Wiley.
Dillman, D. A. (2000). Mail and
Internet surveys: The tailored design method. New York: John
Wiley.
Lam, J. A., Malaney, G. D., & Oteri, L. A.
(1990). Strategies to increase student response rates to mail
surveys. Journal of Marketing for Higher Education,
3(1), 89-105.
Sudman, S. (1976). Applied
sampling. New York: Academic Press.
Williams, E. A., & Anderson, C. M.
(1999). Applications of technology to assist student affairs
researchers. In G. D. Malaney (Ed.), Student affairs research,
evaluation, and assessment: Structure and practice in an era of
change (New Directions for Student Services, No. 85, pp.
61-71). San Francisco: Jossey-Bass.
Wortman, T. I., & Upcraft, M. L. (2001).
Web-based data collection. In J. H. Schuh & M. L. Upcraft,
Assessment practice in student affairs: An applications
manual (pp. 101-125). San Francisco: Jossey-Bass.
Printer Friendly Version
|
|
