Join our mailing list!
©2007
StudentAffairs.com
All Rights Reserved
|
Going Beyond the Conversion of Paper Survey Forms to Web
Surveys
Steve
Chatman
Director
Student Affairs Research and Information - UC Davis
Posted February
15,
2002 Student
Affairs Online, 3 (Winter)
The greatest challenge facing those who would use
the Internet as a survey administration tool is to realize that this
communication medium is capable of supporting fundamental changes in
the way surveys are created, administered, analyzed, and reported.
When student affairs researchers experience the immediate advantages
of Web surveys, they are quick to embrace the technology and too soon
to be satisfied by putting a digitized face on an old friendly form.
This paper goads the researcher to explore and experiment in the use
of this new powerful tool for university surveys of student
populations.
Are Web Surveys as Effective as Paper Equivalents?
At the 2000 California Association for
Institutional Research Conference, research was presented that
directly compared Web and traditional paper surveys conducted across
California universities and concluded that Web surveys produced
equivalent information more efficiently. [1]
And while results have occasionally shown differences between
e-mail/Web surveys and postal/paper surveys, [2]
it is increasingly clear that e-mail contact and Web surveys are
displacing paper-based methods for studies of on-campus populations.
Nagging doubts that evaluators and researchers have about the
introduction of systematic error sources, especially coverage and
nonresponse, tend to be assuaged by decreased administration costs,
comparable response rates, more efficient data management, and
greatly reduced collection time. For these reasons and more,
institutional researchers and nationally prominent higher education
research offices [3]
are converting paper questionnaires to Web forms and dramatically
increasing the number of new surveys delivered by the Web. The
rationalization is something along the lines that the sampling and
nonsampling errors of email contact and Web survey administration are
no worse than those of paper surveys administered via traditional
postal distribution and that other comparative factors clearly favor
Web surveys.
So, what is wrong with performing much more
efficiently - obtaining equally useful information faster and at less
cost? What is wrong is that institutional researchers are too quickly
migrating traditional paper forms to Web versions and are too readily
developing new Web survey forms that look much like traditional paper
forms. They seem to be blind to new possibilities and are instead
assimilating new capabilities into the same old regular processes.
Perhaps a more dramatic change in process is prohibited by higher
education's recent history of working well behind the leading edge.
After all and with rare and notable exceptions [4],
institutional researchers have largely ignored major advances in
using new technologies, especially computer adaptive telephone
interviews.
Institutional researchers are not alone in
assimilating this tool into the same old toolbox and using it to
build information resources in pretty much the same old way. This
assimilation of Web technology to standard survey processes is
illustrated in Dillman's Mail and Internet Surveys [5]
in chapter 11, "Internet and Interactive Voice Response Surveys,"
where Dillman extends TDM principles to the new medium. Dillman does
recognize some modest new capabilities for Web surveys in item and
form design and administration: better managed branching, pop-up
instructions, floating windows for directions, drop-down boxes for
long lists, screening questions to produce a tailored survey on the
fly (more sophisticated branching), animation, video and audio; but
is less helpful regarding significantly different new possibilities
in administration, analysis and reporting.
Remember
the joke about the Aggie, convinced by the salesman to buy a chain
saw because of the huge productivity increase possible, who attempted
to return the chainsaw for full refund because he was sawing less
wood than before? Well, the salesman could not imagine why that might
be the case. He checked the fluids, wiggled the spark plug wire and
gave the cord a good pull. The saw sputtered and came to life causing
the Aggie to exclaim, "What's that noise?"
This
paper offers a more radical viewpoint than Dillman by suggesting
this: within the university environment where institutional research
surveys are most frequently self-administered paper forms with
optically-scanned response sheets mailed to a random sample or
distributed in classrooms, very nearly every aspect of the
development, information collection, analysis, and reporting
processes can be accomplished more effectively using digital
exchanges. The observations made are apropos to universities or other
closed systems where Web access is ubiquitous and the email
population frame is inclusive. This paper asserts that conversion of
paper surveys to Web-surveys is movement in an off direction - that
instead, it is time to rethink information collection processes and
question each traditional practice. It is, in fact, arguably true
that traditional paper forms and survey methodologies in university
environments will soon be anachronisms. After all, survey research is
a type of social communication and Web administration is the new
medium. The possibilities are far reaching.
The
ideas presented in this paper are offered to encourage creative
applications and applied empirical work. Some of the ideas are in use
at universities today, others may not be practical for a few years or
until significant support is available. For an excellent description
of current Web survey types and practices, the reader is referred to
Mick Couper's Public Opinion Quarterly article [6]
but do not expect current applications to be state of the art in this
medium any more than you expect this year's hot new Web authoring
tool to remain the standard for long.
Development of Form and Planning
Process
New
Item-Types
Some
of the new item presentation types now available to university
researchers were mentioned earlier in the paper: [7]
drop-down boxes for long lists, animation, video and audio; but
these ideas can be expanded upon in both simple and more complex
ways. Item design is clearly an area where the university
researcher is limited more by imagination and resources than by
technological capabilities. Other simple extensions of old ideas
to a new medium include use of sliding indicators to allow
respondents to more precisely locate response along continuums and
the ability to absolutely locate response in two dimensions as is
now done categorically during analysis of "ecosystem" responses
(i.e., importance of and progress made toward items). However, the
researcher need not stop with these.
Imagine
items where the respondent can select mode of preferred
presentation (text, audio, video clip), can elect to respond as
they choose (i.e., selection among fixed alternatives, open-ended
text, spoken comment), always have immediate access to pertinent
supporting materials (i.e., a description of the project, item
definition and context, Human Subjects Review Board materials),
and can offer clarification if they believe it to be necessary.
Imagine further that the survey supports multiple languages and
that handicapped students can use their digital aids to be part of
the research process. Would analysis of results from these
variations be more difficult and possibly require new statistical
treatment? Yes. Would responses be more useful? Maybe. Let's
experiment and learn.
New Possibilities
for Item Linking
Researchers
were quick to recognize that the digital media controls can manage
conditional branching better than paper forms but researchers may
not as quickly imagine the extent to which condition-based options
can be employed. When accustomed to the use of connecting lines
drawn between items or short phrases to direct responders to the
next step, it is tempting to be easily satisfied with the
precision with which this is managed in a Web survey, but do not
stop there. On a traditional paper form, anything more than
rudimentary conditional relationships quickly creates a
potentially damaging level of confusion and source of error. In
contrast, routing options under the control of computer coding can
be accurately managed whether the branching is from one item
option to a subsequent item or set of items, or from a set of
responses to several items to a tailored survey administration, a
routing survey within a survey. It should also be clear that
respondent characteristics known from university operating systems
can be used to control questionnaire administration
unobtrusively.
Examples
of routing control come from a UC Davis summer session survey
administered in 2000-01, a housing survey under development, and a
single-item "poll". The first three items of the summer session
survey asked about prior experience with summer session, whether
the student was planning to stay in Davis this summer, and whether
they planned to take UC Davis classes over summer. If they were
not planning to enroll in summer sessions, they were asked whether
financial aid, at current level awarded, would make the
difference. If they had prior experience, they were asked to list
that experience. If they were planning to enroll in summer
sessions, they were asked about the level of student services they
wanted to receive. The branching paths were simple but as the
conditions were not mutually exclusive, it would have been
difficult to manage to direct the desired branching on paper. A
second example comes from a residence survey under development.
For students who changed residence in the past year, the survey
presents a list of attributes by which they rate their residence
this year and last. For each attribute where their current
residence is more highly rated, they are asked how important the
difference was in their decision to move. A third example also
comes from the residence survey. Students are asked how much they
expect to spend for rent and whether they expect to have a private
bedroom. Based on these two variables, students see a series of at
least three of nine apartment floorplans and are asked to choose
their preference and explain why. The selections will be used by
the UCD Greenhouses Project Design Committee guide development of
a housing project to be constructed by 2005.
Using the Web and Email to Create a
More Inclusive Development Process
The same attributes that favor email and Web
delivery of information generally (e.g., fast, wide access,
structured, changing/dynamic) can be effectively used in the
survey development and planning process to keep those involved
with or interested in the development process up to date regarding
suggestions and changes and to greatly expand the number of people
involved. During development, the Web can be used to display item
formatting options, present draft items for comment, offer a
proposed description of the research methodology and logic, give
access to recorded committee discussions, and for many other
applications. In addition, the Website URL can be shared in email
communication to principals representing campus constituencies who
can, in turn, forward the address to others. Using this dendritic
distribution strategy can quickly involve large numbers of
faculty, staff and administrators, all of who need only reply to
the originating author to share comments. In addition, changes can
then be made to the research plan and to items so that the
development process more accurately reflects a dynamic exchange
and incorporation of comments.
A more focused appeal can also be directed to
the campus community by local constituent purview or to remotely
located colleagues. An example of limited local constituent
inclusion for a larger survey including a few campus climate items
would be to direct the attention of those with special interests
or responsibilities in diversity and campus climate to the
pertinent survey items when asking for comment and suggestion. By
doing so, their expertise can more easily improve the survey
process without the need for additional committees or their
inclusion on a much larger, survey-wide, committee. The researcher
can also invite comment by a much wider professional audience by
sending the email appeal to colleagues with absolutely no concern
about distance or location. Colleagues at sister campuses whose
experience might be pertinent should obviously be included but so
to can the opinions of remote colleagues. Yes, it is true that all
this can be done using paper forms, but it is much more difficult
and demands far more resources. Should the survey project director
ever meet face-to-face with campus constituents? Of course, but
those meetings can probably be much smaller, more focused, less
frequent, and more productive.
Pilot Test and Empirical Item
Development
Pilot testing, we know that it's good for us,
so why do we do it so seldom? Is it because it is so much work,
because access to respondents is difficult, or because we just
don't have time? Whatever the reason, pilot testing remains good
practice and with Web surveys and email, it is easier and faster
to accomplish. Because response to email appeals occurs quickly if
at all, a pilot-test can be done in as little as three or four
days. In addition, there are other means by which to test items.
For example, many campuses have some type of polling application
that can be enlisted as can be various types of volunteer email
panels, or email sent to randomly selected students with survey
items attached or presented with at a linked URL.
An extension of these ideas is planned by the
University of California SERU21 (Student Experience at a Research
University in the 21st Century) project team. [8]
As part of the development process, several electronically
supported discussion forums (chat rooms, bulletin boards, mailing
lists, and campus polls) will be used to share possible items with
target group representatives. Once again, the uncoupling of time
and place from communication will support inclusion of students
from across the system and could as easily expand across the
country. Actually, once the researcher leaves the linear sequence
of events model, it is easy to see that the process can be more
dynamic generally - that early survey results can be treated as a
pilot test and can be used to fix, improve, or redirect the
project even if collection is underway. It is analogous to being
able to reach out and change paper survey forms while they are in
the mail or sitting on a desk waiting to be completed. An example
will be offered under the Administration heading.
Administration
Experimentation
The most significant advantages to university
institutional researchers of digital administration can be
summarized in three words: research, research, and research. It
has never been easier to systematically vary survey processes and
assess the consequences of having done so. Is pre-notice email
helpful? What about enticements mailed with pre-notice or promised
rewards sent in pre-notice? Does the subject line matter and if
so, should it be formal, funny, or challenging? What about the
name and title of person sending the mail? What about use of
personalized salutation in a digital administration? OK, so
knowing the answer to these questions for your campus will not
lead to a Nobel Prize. The answers are, however, important and the
opportunity exists to rise above parochial practice, local
anecdotes, and opinion. In past years, the researcher might be
forgiven for electing not to perform the work required to inform
these decisions due to the difficulty and expense of doing so
using traditional tools. No more. It is far too easy to randomly
assign cases to treatments and evaluate results in digital survey
work.
An example comes from the Davis Quality of
Educational Experiences Questionnaire (Davis QE2Q) undergraduate
census administration in spring 2000. The following example
describes an "experiment" conducted on the fourth e-mail contact.
Six subject lines were randomly assigned to non-responders with
each subject line going to 1,600 or more non-responders. The six
subject lines were:
- We know who you are but ….
- Don't let others speak for you.
- UCD students have received over $1,500 in CASH and $1,000
more will be awarded in the next 2 weeks.
- Join the 7,000+ students who have invested 10 minutes to
improve UC Davis
- Who cares what you think anyway? ;^)
- What do we have to do to get you to respond?
The message's body text of the six variations
was identical and all students were contacted on the same day.
Response rates to the six were 8.5%, 9.1%, 8.9%, 6.9%, 9.8%, and
17.2%. Results show that the 6th subject line was
nearly twice as effective in encouraging response. Something about
the directness of wording and the many ways in which a reader can
interpret the prompt proved very useful. The other five were
similarly effective, or ineffective if you prefer. Appeals to
humor, reward, independence, and veiled suggestion of
accountability were not as useful and the appeal to peer behavior,
"Join the 7,000 …" elicited the fewest replies. The sixth
subject line was used with the fifth and final mailing sent to
non-responders who had received one of the other five versions
where it again proved instrumental in the survey reaching a 53%
unadjusted response rate. In fact, the fifth appeal was more
effective than the fourth.
Inexpensive to Ridiculously Cheap
Incremental costs of increasing sample size in
digital administrations are so low that researchers in university
environments will increasingly use very large samples or census
administrations. They will also use ombudsman instruments with
large item pools distributed over several parallel forms to serve
a variety of interests while collecting sufficient detail to
report at low levels of aggregation. Whether administered to a
sample of 100 or 1,000 or to a population of 10,000, there are
only two clearly potential negative consequences to the researcher
of greatly increased sample sizes. First, the increase in number
of remarks to open-ended comments is directly proportional, and
unless cut-off or limited by open-ended items going to a smaller
subset, the volume of material becomes overwhelming. The second
potential negative is an obligation that researchers accept when
the decision is made to greatly expand the number sampled. That
obligation is to justify the intrusion and collective expenditure
of time and effort by respondents. If that justification is that
you will provide results at lower levels of aggregation then you
must fulfill the promise.
Mass Media and Cultural Campaigns
A byproduct of very large administrations is
the possibility to encourage response through a coordinated
campaign for participation. As the sample becomes increasingly
inclusive, it becomes reasonable to use commonly directed mass
appeals. As an example from Davis QE2Q, a census survey of the
undergraduate population, public appeals included a campus
newspaper story published just before the start of collection and
weekly paid advertisement in the same paper announcing time
remaining, the winner of last week's drawing for $500, and the
winners from earlier weeks. Other appeals included posted signs,
table-top advertisements at the student union building, staff
presence at a table near the most popular on-campus lunch
facility, and, in future administrations, will include notice at
the campus Web-portal interface. In addition to mass appeals,
email communications were directed to associate deans for
undergraduate education and to area and ethnic study offices
asking that they use local email mailing lists to encourage
students to participate. Other affiliated groups whose support can
be sought are student organizations and academic or social
clubs.
Targeting the
Campaign During Collection
Reiterating
an earlier theme, the survey process need not be linear sequential
with the researcher following a long series of predetermined
steps. Instead, the process can be more spiral shaped. For
example, the composition of respondents after the second or third
contact can be used to tailor other tries. If males are
underrepresented or if minority students are disproportionately
underrepresented among respondents, then future appeals can be
directed at subpopulations.
One
extreme example comes from the Davis QE2Q project. Results from
the first week of data collection identified an extreme anomaly.
There were no first-time student respondents. That observation led
to discovery of a programming error, creation of a supplemental
file, and distribution of a mailing directed at first-year
students - all accomplished within 24 hours from discovery of the
error. By the third mailing, a single contact schedule was used
for both groups because response rates by student class level had
equalized. Imagine discovering that error during data cleaning of
optically scanned forms months later and after students had left
for summer.
Nested Items
and Follow-Up Possibilities
Another way in which analysis of early results
can be used to improve the survey process is by allowing the
researcher to seek clarification even when the need for
clarification was not anticipated before the survey was
distributed. If early responses are ambiguous or if the results
are unclear because of weak item design, it is possible to email a
subset of the population to ask for clarification or elaboration
and to modify the survey by replacing or supplementing existing
items. While UC Davis has not used this strategy on a large-scale
survey administration, it has been part of a volunteer panel
survey process. In sum, it should be clear that the distinctions
between pilot-testing, administration, and follow-up will become
increasingly blurred.
Advantages of Email
Email offers communication advantages at many
different levels: less costly, easier to produce, faster from
development to collection, easily forwarded, supports inclusion of
Website link, et cetera. It is also time-stamped and many mail
distribution programs support personalization and maintain a
tracking record. These advantages all fall on the side of the
survey administrator. There are also potential benefits for the
respondent, most of who are very familiar and comfortable with
email exchanges because they can more easily ask for help,
clarification, or other assistance. It may be standard practice to
include a contact person with survey appeals (name, address, phone
number), but locally it has been very rare for the contact person
to receive inquiries with standard mail contact. The informality
of email seems to encourage dialogue and results in a few
exchanges with respondents.
For example, each of the Davis QE2Q appeals
included the project director's email address and phone number and
while there was never an onslaught, there was a constant trickle.
On days when thousands of email messages were sent, a dozen
students would take advantage of the easy access. Most of these
contacts were either requests for help or to be removed from the
mailing list, but others were very interesting inquiries or
opportunities for students to share strongly held opinions.
Frankly, it was a chance for some students to vent anger and
frustration.
Given the modest volume it was not difficult to
answer each and every inquiry personally: to direct some students
to other services, to invite students having problems to come by
the office and help us fix the bug, and at least some
non-respondents became participants as a result. A fun example was
the student who responded to the appeal, "What do we have to do to
get you to respond to this survey?" with the answer, "Pay me." To
which SARI replied by asking, "How much?" and a couple of emails
later found the student promising to complete the survey without
special compensation. While statistically inconsequential, this
exchange and others were clearly helpful and often fun for the
student and project director. In other cases, the communication
was more significant. There were a few students who encountered
problems gaining access due a programming limitation. Because of
the ease of communication, we soon learned of and were able to fix
this problem. Those who contacted SARI were invited to come by the
office where they were shown appreciation and received a $10 gift
certificate for bringing the problem to SARI's attention.
Packaging
There
are several practical options available to the researcher
regarding composition of the survey form and the size and number
of samples. Some of these include the use of single-item formats
(polling), short forms (both stand-alone and intercept surveys),
single purpose versions (matriculation, alumni evaluation), and
composite forms, including ombudsman survey compilations. Each of
these can be single or multiple forms. The possibilities are
numerous and are available using traditional materials, but are
much more easily managed with digital survey processes. To
summarize, a single long Web survey form can, with very little
effort, be presented as a series of individual items, a collection
of short survey forms, or even just one part of a much larger
collection effort. This was done on the Davis QE2Q survey where
respondents received one of five forms. The five forms shared an
academic component and university service, campus climate, and
satisfaction items were randomly distributed except that similar
measures, satisfaction with amount of financial aid and
satisfaction with financial aid services for example, always
appeared together.
Regardless
of the packaging selected, there are decisions to be made about
the joint relationship of sample and items. Should items be
randomly distributed over students? Should several student samples
be taken, each presented with part of a larger number of items?
Should all students be asked all items? Obviously, the answer is
that it depends, as it should, on the purpose of the project. It
should not depend on the format of the survey and one approach is
not always best. Administration in a digital format facilitates
use of a wide variety of possibilities for experimentation. A
residence survey under development illustrates how respondent
selection can serve multiple purposes and help establish the
importance of commonly held notions about randomization.
The
Greenhouses Design Committee elected to use a random sample of
1,000 sophomores, juniors and seniors. However, they also decided
to open participation to all students and to offer a few
significant prizes. Will the characteristics and responses of
volunteer participants be different from those of randomly
selected, representative, students? We will soon know. The
important point for consideration here is that we can now
determine the impact of respondent initiated participation. The
use of a highly visible Web site address and a modest security
gate permits openly broadcasted appeals for those interested in
the topic or the prizes.
Controlling Design Error
The digital medium greatly facilitates
randomization and its use in experimental or quasi-experimental
designs. Items can be randomly sequenced to prevent order effects,
items can be randomly assigned to respondents or respondents to
items, and randomization can extend to those factors being
systematically varied as mentioned earlier (e.g., pre-notice
variations, enticements, etc.). One capability that holds special
promise for future applications is control of measurement
precision, or more accurately, item administration until the
required level of precision is obtained. Standard error of
measurement is a function of sample size and variance. If an
acceptable level of precision can be predetermined, then there is
no obvious reason to present the item to participants once that
level is attained. Inversely, there is ample reason to continue to
administer an item where the required standard error of
measurement has not been reached. If SEM were established for
demographic or other levels of analysis, then it is also possible
to selectively administer the item to that subpopulation where
additional data are required. In other words, every reasonable
effort should be made to respect the time and effort asked of the
target population and this can be best accomplished by established
acceptable precision by item by respondent group of interest. This
is in contrast to standard practices assuming maximum variance for
all items.
Publication Advantages of Web Reporting
Some of the more interesting possibilities for Web
survey work are in presentation of results where previously
unavailable detail and statistical support is possible. A Web
interface to the Davis QE2Q survey results will be used as an
example. The Davis QE2Q was a census survey of the undergraduate
population that managed a 53% response rate. The survey was designed
to assess a variety of issues but especially to support the
presentation of academic and instructional information at the level
of the academic major. While on one hand, this level of detail should
be more effective in influencing the behavior of faculty and academic
administrators, the other hand can not be seen because it is buried
under a mountain of data tables. The cross-tabulation of items by
majors by demographic variables of interest (e.g., sex, class level,
race/ethnicity) produces several hundred thousand figures - a
prohibitive amount for paper reporting to be widely distributed, but
less of a problem for an interactive Web tool. Stated more precisely,
the cross-tabulations could be printed on paper but distributing the
voluminous results to the campus community would be prohibitive and
wasteful. A Web interactive tool is well suited to perform the task
and the first version of this report generator is now running as an
analytical tool at www.sariweb.ucdavis.edu\DavisQE2Q.
This example of the Davis QE2Q analytical tool is
only a first step and breaks no new delivery ground - others have
used Web variable selection to take users to a particular set of
results. The difference is more along conceptual lines because the
information is being generated on the fly from the original data set,
but that is only a start. Among interesting applications that a more
innovative presentation system might include are:
- statistical tests and selected comparisons on demand and on
the fly,
- user choice of presentation modes (i.e., graphic, tabular,
descriptive),
- access to respondent clarifying remarks,
- attached comments by concerned institutional parities (i.e.,
department chairperson reactions),
- participant access to their own entries with comparison to
group results,
- better use of epistemological principles and visual interest
by forced choice interaction (prediction followed by observation),
slow display graphics (watch bars grow, points plotted, or a
central tendency indicator move along scale and wonder where it
will stop), and
- an interface constructed to support user entry of ID sets to
view aggregate results for that group (e.g., campus recreation,
peer counseling, fraternities, academic organizations, and
others).
One
other possibility will be mentioned because it illustrates that more
accurate presentation of results is possible using dynamic displays.
Most researchers are familiar with rank-ordered lists and the undue
importance assigned rank position as if relative position could be
established absolutely. A more accurate presentation would
incorporate random variation and error of measurement associated with
the ranking value to produce figures for each record. The records
could then be rank ordered by the predicted value. The resulting
rank-ordered list could differ for each viewer - more accurately
reflecting the lack of difference between nearby entities. For
example, if overall satisfaction with instruction for academic
divisions A and B were not significantly different, but A had a
slightly higher mean than B, then most people would see A appearing
before B in rank-ordered lists but nearly as many would see B before
A. In this way, differences that are not statistically discernible
can be displayed in a way that helped to prevent undue importance
being assigned to relative position. Just as moving a paper survey to
a digital format can be movement in the wrong direction, so too can
be simple publication of static material in a digital format capable
of much more.
The Future is
in Banking
Where
might survey research be headed? One possible answer is toward item
banking and automated presentation of a type analogous to
developments in computerized adaptive testing (CAT) developed 20
years ago and commonly used today. Oversimplified, CAT uses item
characteristics and an individual's prior responses to select the
next item among those remaining that provides the most information
about an individual's ability and continues the process of selecting,
administering, and scoring until an estimate of ability can be
determined with an acceptable level of precision. It is an efficient
and effective approach that suggests analogous survey strategies.
In
survey administration, the common process is typically one of
deriving sample-level parameters with which to predict population
characteristics and the banking analogy would be to administer an
item when and where needed until acceptable population parameter
precision is attained. The core item, its "form", might reside in
HTML with required ASP or ColdFusion code to support the full
complement of associated steps from selection by the developer, to
administration, to reporting.
Imagine
the researcher selecting among alternative items stored in a bank.
The selection might be based on content, prior performance, and the
availability of appropriate norms. The researcher might then specify
administration parameters (target population or populations,
acceptable SEM precision, collection interval, and delivery mode).
The resulting set of items and design parameters become the form and
administration plan. Items would be administered according to
specification until the preset acceptable level of precision was
reached. The design and collection phases would be over. Analysis and
reporting would come next.
Much
of the quantification would have been ongoing and automated, and the
final results would be accessible through a user interface and would
also contribute to the historical record for that item. In this
model, construction, administration, analysis and reporting are
linked to the central item bank. Information stored with each item
would likely include content area, delivery mode, links to other
items through scale membership or parent/child relationships for
conditional structures, history of use, HSRB approval, comparative
norms, and results by date (including clarifications) along commonly
used reporting categories.
Summary
There is increasing evidence that Web surveys can
produce results comparable to traditional paper instruments and that
they can do so faster, more cheaply, and with fewer coding errors.
These are good reasons to switch from paper to Web forms, but the
question of how to put a paper survey on the Web might be the wrong
one to ask. Better questions to ask about the use of this medium
include those that follow.
Can we use this medium to learn more and different
information from our students?
Can we be more responsive to subject preferences and better
support elaboration?
Can we better control design and survey administration
effects?
Can we improve information delivery and increase the likelihood
that results will be used correctly and effectively?
Affirmative answers to these questions suggest
that survey research can and should enter a new phase.
This paper has shared some ideas, guesses if you
rather, about future survey engagement and analytical processes in
this new arena. A few of the more simple applications were
illustrated using a variety of results from a recent completely
electronic census survey (exclusively email and Web-entry) of a large
undergraduate population, a pre-recruited panel, a polling
application, and others will be used in the SERU21 project. However,
none of the material presented absolves the researcher of the
obligation to produce good instruments that appropriately cover the
material of interest, to sample according to intended use, to
struggle to control sampling and nonsampling errors through proper
survey administration, and to then communicate those results through
good analysis. Additionally, the techniques and strategies suggested
by this paper are only appropriate to populations with near universal
access to email and the Internet - universities today and the general
population tomorrow. It is an exciting time in which to do university
survey work.
[1]
Daly, B., Thomson, G., & Cross, J. (2000). Web vs. paper
surveys: Lessons from a direct large-scale comparison. Paper
presented at the Annual Meeting of California AIR.
Printer Friendly Version
|
|
