Since the beginning of quantitative social science, a great deal of research has been done using questionnaires. The structuring, wording, and ordering of questions has traditionally been viewed as an art, not a science, best guided by intuition. But in recent years, it has become clear that this is an antiquated and even dangerous view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. Intuition often leads us astray in the questionnaire design field, as becomes clear when putting intuitions to the test via scientific evaluation. A large body of relevant scientific studies has now accumulated, and when taken together, the findings point to a series of formal rules for how best to design questions. Yet the vast majority of contemporary questionnaire design fails to follow these rules, because they are not yet widely understood. During the past 25 years, Jon Krosnick has been gathering up the huge body of evidence on optimal questionnaire design for this forthcoming book, The Handbook of Questionnaire Design. In this course, Dr. Krosnick will review the contents of the book.
How could this body of scientific knowledge have been overlooked for so long? The reason is that it is scattered across the publication outlets of numerous disciplines, and this literature has never been comprehensively gathered and integratively reviewed in a central place. Doing so has been a principal project for Jon Krosnick during the past twenty years. In doing this work, he has discovered the emergence of general principles of optimal questionnaire design that are often at variance with common practices in questionnaire construction. And in many cases, these departures in practice have reduced the accuracy of collected data, even though other question forms could easily have been employed instead that would have avoided bias and improved precision of measurement.
Because of the complexity of this literature, it does not yield a short and simple list of rules to guide every decision to be made by a questionnaire designer, each supported by a few documentary references, and each obviously justified by all relevant studies. The issues addressed are multifaceted, and many are still in the process of being resolved by innovative and creative new research. But there is a great deal of richness in the existing literature that provides useful guidance for scholars interested in maximizing the reliability, validity, and efficiency of the measurement instruments they employ in their research. Reviewing this literature and its implications is the focus of the proposed workshop.
Goals of the Workshop/Seminar
The workshop would serve many purposes. First and foremost, it would educate participants about the optimal techniques for questionnaire design, for guarding against measurement artifacts, and for analyzing data in order to overcome the biasing impact of such artifacts. And because the huge accumulated literature on questionnaire design does not provide guidance to researchers about how to handle every decision they will face, the second and equally important goal of the workshop is to teach participants a new way of thinking about questionnaire design, rooted in psychology. The aim is to get researchers into the heads of their respondents in a way that few have done before. By gaining insight into how respondents approach their tasks of interpreting questions, searching their memories for information, integrating that information into judgments, and expressing those judgments in words, workshop participants will begin to develop skills that will help them to mange design issues for which there are not yet formal rules.
In addition to helping participants to design better questionnaires, the workshop will equip analysts of questionnaire data to look from a new perspective as they evaluate the meaning of their findings, knowing how measurement artifacts can lead them astray.
The workshop would be engaging and useful for participants with a wide variety of backgrounds. In the academic world, questionnaire designers and analysts of questionnaire data can be found throughout the social sciences. Such scholars work in political science, psychology, sociology, economics, communication, health, business, law, journalism, anthropology, the health professions, business, and many other disciplines. Individuals from all of these areas of the field would find this course useful for improving their work.
Outside of academia, researchers at marketing and advertising firms, survey research firms, the news media, consulting companies, the research arms of government, and research departments of manufacturing and service corporations all routinely design questionnaires and use questionnaire data.
In short, whether a researcher uses questionnaires in laboratory experiments involving 50 participants or in large-scale representative sample surveys of tens of thousands of respondents or simply reads and interprets questionnaire-based data collected by others, this course is intended to help him or her do better work.
During twenty years of experience teaching this workshop, it has become clear that participants benefit no matter what their level of expertise and experience with questionnaire design and data analysis. For people who are new to questionnaire design, this can be an introduction to design issues they had never thought much about before. For people with a great deal of questionnaire design experience, the workshop challenges many of their long-standing assumptions and energizes their high-level thinking about the questionnaire response process, often leading them to become leaders in advancing our growing understanding of this phenomenon.
Summary of Workshop Content
The workshop content is organized into lectures that address 13 primary topics, generally organized according to the practical decisions a designer must make when putting a questionnaire together. The first session will provide a general introduction to the new view of question-answering that forms the foundation of the course. The second session will discuss the first decision a researcher must make when writing a question: will it be open-ended (allowing respondents to answer in their own words) or closed-ended (requiring respondents to select an answer from among an offered set)? The next five sessions will discuss the formatting decisions that must be made for closed questions: should a rating or ranking format be used, and, for rating questions, how many scale points should be offered with what verbal labels and in what order? The remaining sessions will discuss issues equally applicable to closed and open questions. Session 8 will address the question of non-attitudes: what evidence is there that respondents construct opinions on the spot rather than simply reporting pre-formulated views, what evidence is there that respondents provide meaningless responses, what can be done to effectively manage these issues? Session 9 will review numerous non-structural aspects of question wording that can have effects on responses. Session 10 will discuss effects on responses of the order in which questions are asked. Session 11 will discuss the merits and flaws inherent in questions asking people to recall attitudes or beliefs that they held at some previous time. Session 12 will discuss the strengths and weaknesses of questions asking people why they feel a certain way or why they behave in a certain way. And finally, Session 13 will review and summarize the primary conclusions of the prior sessions.
Listing of Session Topics for the Workshop
1. Introduction: The Survey Response Process
2. Open versus Closed Questions
3. Closed Questions: Rating versus Ranking
4. Closed Questions: Number of Scale Points
5. Closed Questions: Verbal vs. Numeric Scale Point Labels
6. Agree/Disagree, True/False, and Yes/No Questions: The Problem of Acquiescence
7. Closed Questions: Response Order Effects
8. “Don’t Know” and “No Opinion” Response Options
9. Question Wording
10. Question Order
11. Social Desirability
12. Attitude Recall Questions
13. Asking "Why?"