Experimental Design
In the past twenty years, experimental research designs have become increasingly popular in disciplines in the social sciences, such as political science, that do not have a longstanding tradition of experimentation. Advocates of experiments argue that these methods are the gold standard in assessing causality: whether a variable of interest is actually caused by another variable. However, critics push back that causality in the context of an experiment does not always translate well to causal processes in the real world.
There is no single research design that is perfectly suited to test every hypothesis a social scientist might assert: the best research matches the most effective research design to the particular puzzle to be explored. The purpose of this module is to help you understand what differentiates experiments from other forms of research designs to facilitate your informed evaluation of the benefits and drawbacks of experimentation, in order to assess whether an experiment is well-suited for your research question.
By the end of this module, you should be comfortable with the following terminology: fundamental problem of causal inference, treatment, control, random assignment, internal validity, external validity, and convenience sample. You should also be aware of the avenues open to you to conduct experiments for an independent research project.
View Module as PDF
Background
1. What is an Experiment?
In common parlance, people often say “experiment” when they mean “study.” All experiments are studies, but not all studies (in fact the vast majority of studies) are not experiments. In social science research methods, labeling a study as an experiment means something very precise.
Morton and Williams (2010) write that an experiment is conducted when “a researcher intervenes in the data generating process by purposefully manipulating elements of that process” (p. 42). Researchers want as much control as possible over their manipulations to minimize the interference of confounding factors. The most common way of controlling confounding factors is to compare experimental results to a baseline where all observable conditions are identical except for the presence of the manipulation. In within-subjects designs, subjects serve as their own baseline: all facets of the study are held constant and the same subjects experience all manipulations. However, between-subjects designs are more versatile and more common, and the selection of an appropriate baseline comparison, a control group, is essential.
Consequently, when most political scientists use the term “experiment,” they refer to a study where subjects (or whatever entity we are studying) are randomly assigned to different treatments, where the treatments are causal interventions. Druckman et al. (2011) write “In contrast to modes of research that address descriptive or interpretive questions, researchers design experiments to address causal questions. A causal question invites a comparison between two states of the world: one in which some sort of intervention is administered (a treated state, i.e. exposing a subject to a stimulus) and another in which it is not (an untreated state)” (pg. 16).
2. When are Experiments Most Appropriate?
When is an experiment the optimal design for a research question? First, if you are interested in studying causation, an experiment is the cleanest way to show a causal relationship between two variables. The biggest problem plaguing many observation studies is assessing whether one variable (the independent variable) actually causes another variable (the dependent variable). For example, does television news actually influence public opinion? It could be the case that people likely to watch television news differ in important ways from those people who do not, so in an observational study, we cannot pinpoint the role of television news in actually changing opinion (Morton and Williams 2010, p. 13). This kind of question is well-suited to an experimental design.
Second, if you are interested in human behavior, experiments often provide the most straightforward way to understand the mechanisms that explain why people do what they do. Demonstrating a relationship between two variables in observational data does not tell you how that relationship actually operates, while experimental techniques can isolate the process of influence.
Finally, some research puzzles are more conducive to experiments than others. We would never want to randomly assign individuals to experience civil war, nor would we want to randomly assign some people to be disenfranchised, for example. However, a special kind of study, natural experiments (described more below) can help researchers leverage the concept of random assignment even if they are not the force actually assigning the treatment. A creative experimentalist can design a treatment to ethically test a wide variety of political and social phenomenon. In recent years, there have been papers published in political science journals utilizing a wide variety of experimental stimuli to test questions like a “get out the vote” experiment on 61-million Facebook users.
Causal Inference: Key Concepts
3. Fundamental Problem of Causal Inference
If causal questions involve comparisons between two states of the world, the fundamental problem of causal inference declares that we cannot simultaneously observe those two states of the world, one in which an individual is treated and one in which the individual is not. We cannot directly overcome this reality but we can work around it by using random assignment. Random assignment means that “each entity being studied has an equal chance to be in a particular treatment condition” (Druckman et al. 2011, p. 16-17). While we expect there to be random sampling variation (we don’t expect the two treatment groups to be exactly identical on all characteristics), this variation is due to chance alone, meaning it is uncorrelated with the treatment subjects receive. Except for the intervention itself, we can assume that randomly assigned groups are probabilistically equivalent. In other words, we can assume that the control group acts like the treatment group would have if the treatment group had not received the treatment, allowing us to assess the average treatment effect.
4. The Meaning of “Control”
Morton and Williams (2010) define control in experiments as a researcher fixing or holding constant elements of the data generating process in an experiment to better measure the effects of the manipulations (treatments) (p. 44). This control is done to minimize the effect of observable and unobservable confounding factors in an experiment. Researchers have the most control in a laboratory environment, where they are able to measure more factors to make them observable and hold constant more of the factors influencing how subjects receive the treatments.
5. Random Assignment vs. Random Selection
Although these two terms sound similar, they are distinct concepts. Random sampling refers to the process by which subjects are selected from a population such that every potential participant has an equal chance of being selected for the study. An experiment does not require a random sample, although its external validity can be increased if a random sample is used. Random assignment refers to the procedure for determining how subjects in an experimental study are assigned to treatment groups.
Within and Between Subject Designs: In a between-subjects design, each subject is randomly assigned to a single treatment, and the researcher compares the treatment groups to each other or to the control group. In a within subjects design, individuals serve as a control to themselves, and a subject is observed before and after receiving the treatment. While within-subjects designs are appropriate in many instances, the concern is that there can be many confounding factors with the delivery of the treatment itself, making it harder to establish a causal connection between the treatment and the outcome variable.
6. Internal vs. External Validity
Experiments offer very strong internal validity—the ability to assess the degree to which the independent variable (the treatment you are manipulating) actually causes your dependent variable. The best experimental research designs include appropriate baseline or control groups, so that researchers can make the most precise causal inferences possible. Conversely, experiments are weaker in their external validity, or the degree to which the causal relationship demonstrated in the experiment could be replicated in other contexts, with different people, or with different operationalizations of the treatment or outcome variables.
7. Types of Experiments
Survey Experiments: The term survey experiment refers to a broad category of stimuli where an individual decision-making experiment is embedded within a survey (Morton and Williams 2010, p. 79). These can take many forms. In the most basic form of survey experiments, researchers can vary question wording or question ordering to assess whether these factors affect respondents’ answers. More complicated survey experiments involve vignettes, where subjects read about hypothetical scenarios in which researchers manipulate various features of the situation. Increasingly common are list experiments, a form of a permissive design (Druckman et al. 2011, p. 107) that removes pressure favoring one response over another, allowing respondents to more honestly convey their behaviors or attitudes. In a list experiment, subjects report the number of statements with which they agree, not the statements themselves. Subjects are randomly assigned to one of two groups, a baseline condition with a list of statements of length n, and a treatment condition with a list of identical statements but where an additional, potentially objectionable statement is inserted (length n+1). By comparing the mean number of items agreed with between the two groups, a researcher can estimate the proportion of people who agreed with the objectionable statement.
Lab Experiments: The basic requirements for a lab experiment are that subjects are recruited to a common physical location where the study takes place, and a researcher directs the behavior of the subjects. Lab experiments in political science have their root in the experimental techniques developed in other disciplines. Experimental and social psychology have had an enormous influence on the field of political psychology, and political psychology experiments typically adhere to the norms established in these disciplines. Subjects are paid a flat fee (or given class credit if student convenience samples are used) and researchers assume that subjects will behave sincerely. Researchers often go to great length to make their studies seem naturalistic and may use deception. Conversely, in experiments modeled after techniques used in behavioral economics, subjects are paid based on the choices they make in the experiment under the assumption that the payment motivates subjects to behave in a more natural fashion. These experiments tend to be abstracted away from reality and are used to test theories and formal models. Deception is rarely used. Finally, “lab in the field” experiments occur when researchers bring the controlled environment of the lab into a field environment, in an effort to conduct experiments on a wider variety of populations and to increase the external validity of the results of the experiment.
Field Experiments: Field experiments occur when a researcher’s intervention takes place in subjects’ natural environment. One of the most common types of field experiments in political science is “get out the vote” experiments, where registered voters are randomly assigned to receive an encouragement to vote in some form (mailing, door hanger, phone call, etc.) and the turnout rate of the treatment group is compared to a group that did not receive the encouragement. However, there are many other clever and interesting manipulations that take place in the natural environment.
“Natural” Experiments: The term natural experiment is applied to studies where researchers take advantage of random assignments that occur naturally in the world. One of the most common examples of this is a lottery. Researchers often label their study as a natural experiment even if the events that happen to some people but not others are not determined in a truly random way, arguing that the process that led to the assignment of the “treatment” was administered as if it were random. While these are not true natural experiments, they can be leveraged in meaningful ways to help researchers understand a wider variety of phenomena.
Implementing an Experiment
8. Finding Subjects
All research designs pose unique challenges. One of the biggest in experimental designs is the need to recruit subjects to take your study. In an ideal world, every experiment would be conducted on a random sample of the population of interest to achieve both strong internal and external validity. But one of the main benefits of experimentation is that you can achieve strong internal validity on a convenience sample. Therefore, experiments disproportionately rely on samples that are not randomly selected and researchers make no claim that their sample is representative.
Although reliance on non-probability convenience samples is widely accepted, it is still a considerable amount of effort to identify and recruit subjects to participate in your study. To assist you in this process, the SSRMC implements the Omnibus Project every semester. The project coordinates and streamlines the development of a student subject pool (a convenience sample) for faculty and student research. Project coordinators collect data for a set of common variables, such as demographic information and political covariates, which are provided to all researchers submitting proposals. Individual researchers submit their survey questions and customize their own portion of the survey instrument. The subject pool is comprised of students enrolled in departmental classes; some instructors will require participation or offer extra credit to students to participate in the study.
Another cost-effective option is to use Amazon’s Mechanical Turk (MTurk) service, a website that allows researchers to publish tasks (HITs or Human Intelligence Tasks) and provide payment to subjects who choose to participate. Those who request a task can limit the availability of the task to respondents who meet certain qualifications, such as age or location. Studies using samples from Mechanical Turk have been published in the top journals in political science and have been found to replicate important experimental findings in psychology. While MTurk is still a convenience sample, it is more representative of adult populations than undergraduate samples or samples populated from those who respond to web advertisements.
If you have used the above samples to collect pilot results that are promising tests of your hypotheses, with the support of a faculty member, you can put together a proposal for the Time Share Experiments in the Social Sciences. This project, sponsored by the National Science Foundation, allows researchers to submit experimental proposals for consideration to be fielded on a representative sample of American adults on an Internet survey platform.
9. Delivering the Treatments
First and foremost, studies involving human subjects always require approval from the Institutional Review Board (IRB). Applications to the IRB must include documentation that you have successfully completed the ethics training mandated by the federal government. Depending on the nature of your experiment, your study may be exempt from full review of the board, but that is a decision made by the chair of the IRB, not by you as the researcher. You should allow at least three weeks for your study to be approved before you can collect data.
Political scientists tend to conduct three main types of experiments: survey experiments, lab experiments (including lab-in-the-field experiments), and field experiments.
If you want to do a survey experiment, the simplest option is Qualtrics, a software program for which the College maintains a subscription for both computer-based and mobile platforms (for experiments you want to do remotely). Qualtrics can be programmed in very sophisticated ways to randomly assign subjects to different treatments.
One of the drawbacks of survey experiments that subjects take in the comfort of their own environment is a lack of control on the part of the researcher in controlling that environment. For example, if you are conducting your survey on an Internet sample, subjects are able to browse the Internet or walk away from the computer while they are taking your study. This is most problematic if there is reason to think that some facet of your treatment might make subjects more likely to get distracted or visit other webpages, for example if your treatment is very long or if you ask post-test questions about political knowledge, where subjects might feel inclined to seek out the correct answers online. Programs like Qualtrics have some built in functionalities to be able to detect this.
However, in some instances, you may want to conduct a survey experiment in the laboratory, either to exert more control over the experiment or because you want to deliver a treatment that is not well suited to the online or phone format. In that instance you may want to use the research lab facilities of the SSRMC. If you are doing a lab experiment and you are interested in delivering media (images, audio, or videos) to subjects in a laboratory environment, one of the most popular software programs to do so is SuperLab. The SSRMC currently has licenses for this program on one computer. An example honors thesis that relies on SuperLab for stimulus delivery can be found here.
If you are interested in doing an experiment in the field, academics frequently partner with outside organizations—such as campaigns or advocacy organizations—because it is often difficult to get access to the large subject pools necessary to conduct these studies. William & Mary undergraduates have successfully done this, and this honors thesis is a great example.
10. Analyzing the Data
There are more sophisticated ways of analyzing the data from experiments, and methodologists are constantly developing new ways to extract less biased estimates of the causal effects in a study. While these more advanced approaches are beyond the scope of this introductory module, the resources listed at the end of this module contain more in-depth information.
11. Considerations and Cautions
Studies involving human subjects must go through ethics review for good reason, and this is especially important in the case of experiments, where researchers manipulate the environment or stimuli to which participants are exposed.
Different experimental traditions in the social sciences have different norms. One of the biggest differences is in the instructions that researchers give participants about the nature of the experimental tasks. Experiments rooted in psychology often allow researchers to use mild deception in their instructions if the researcher thinks that knowing the true purpose of the study would alter the way that participants behave. The SSRMC allows deception in studies, as long as that deception is approved by the Institutional Review Board. However, deception is almost always avoided in experiments rooted in economics. A second major difference between economic and psychology experiments is whether (or how) subjects are incentivized for their participation.
The classic tradeoff in experimental design is between internal validity and external validity. While experiments have high internal validity, to varying degrees, they may lack external validity, the ability of a researcher to make claims about how the results of the study would generalize and hold up in different contexts. One particularly common generalizability concern stems from differences in the sample used in your experiment compared the population to which you want to generalize. When is it problematic to generalize the findings of an experiment conducted on a convenience (often, student) population? First, it is important to know how the student population differs from a more representative population. The obvious answers are in age, education level, and geographic location.
But other factors can matter as well. The key question to ask is “how else are college students different in a way that should affect the strength or direction of the causal relationship I am testing?” If your convenience sample is different in a way that makes it harder to find the relationship you observe, then you can assert that your study likely underestimates the relationship between the variables in a more representative population (a testable proposition!) However, if your sample makes it easier to find effects, then generalizability concerns become more serious. Sometimes, these concerns are very large. For example, college students are particularly susceptible to conformity (Sears, 1986), which could be important depending on the nature of your study. The Omnibus Project draws from courses in the government and international relations program, suggesting that our participants have a greater interest and level of knowledge about politics. These factors may help or hurt your ability to make claims about how general your findings are.
More broadly speaking, researchers have commented on the abundance of WEIRD subjects in experimental studies: subjects that are Western, educated and come from countries that are industrialized, rich and democratic. This has been written about considerably in the popular media and has been addressed in academic research as well.
12. Additional Resources
There are many excellent resources available online, through SWEM library, and through the SSRMC research methods collection. The module below draws on information from these sources, but there is much more detail available in the original sources.
Experimental Design
- Druckman, James N. et al. 2011. Cambridge Handbook of Experimental Political Science.
- Rebecca Morton and Kenneth Williams. 2010. From Nature to the Lab: Experimental Political Science and the Study of Causality
- McDermott, Rose. 2002. “Experimental Methods in Political Science.” Annual Review of Political Science 5:31-61.
- McDermott, Rose. 2002. “Experimental Methodology in Political Science.” Political Analysis 10(4):325-342
- Druckman, James, Donald Green, James Kuklinkski, and Arthur Lupia. 2006. “The Growth and Development of Experimental Research in Political Science,” American Political Science Review 100:627-635
- Gaines, Brian et al. 2006. “The Logic of the Survey Experiment Reexamined.” Political Analysis
- Imai,Kosuke et al. 2011. “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review
Field Experiments
- Green and Gerber’s Field Experiments book
- Yales ISPS Field Experiments Initiative as well as their Data Archive