Survey Tools Module

1. Finding Subjects

All research designs pose unique challenges. One of the biggest in experimental designs is the need to recruit subjects to take your study. In an ideal world, every experiment would be conducted on a random sample of the population of interest to achieve both strong internal and external validity. But one of the main benefits of experimentation is that you can achieve strong internal validity on a convenience sample. Therefore, experiments disproportionately rely on samples that are not randomly selected and researchers make no claim that their sample is representative.

Although reliance on non-probability convenience samples is widely accepted, it is still a considerable amount of effort to identify and recruit subjects to participate in your study. To assist you in this process, the SSRMC implements the Omnibus Project every semester. The project coordinates and streamlines the development of a student subject pool (a convenience sample) for faculty and student research. Project coordinators collect data for a set of common variables, such as demographic information and political covariates, which are provided to all researchers submitting proposals. Individual researchers submit their survey questions and customize their own portion of the survey instrument. The subject pool is comprised of students enrolled in departmental classes; some instructors will require participation or offer extra credit to students to participate in the study.

pp_img_3_col_mechanical_turk_good_logo_no_artificial_2378x171

Another cost-effective option is to use Amazon’s Mechanical Turk (MTurk) service, a website that allows researchers to publish tasks (HITs or Human Intelligence Tasks) and provide payment to subjects who choose to participate. Those who request a task can limit the availability of the task to respondents who meet certain qualifications, such as age or location. Studies using samples from Mechanical Turk have been published in the top journals in political science and have been found to replicate important experimental findings in psychology. While MTurk is still a convenience sample, it is more representative of adult populations than undergraduate samples or samples populated from those who respond to web advertisements.

If you have used the above samples to collect pilot results that are promising tests of your hypotheses, with the support of a faculty member, you can put together a proposal for the Time Share Experiments in the Social Sciences. This project, sponsored by the National Science Foundation, allows researchers to submit experimental proposals for consideration to be fielded on a representative sample of American adults on an Internet survey platform.

2. Delivering the Treatments

proRev

First and foremost, studies involving human subjects always require approval from the Institutional Review Board (IRB). Applications to the IRB must include documentation that you have successfully completed the ethics training mandated by the federal government. Depending on the nature of your experiment, your study may be exempt from full review of the board, but that is a decision made by the chair of the IRB, not by you as the researcher. You should allow at least three weeks for your study to be approved before you can collect data.

Political scientists tend to conduct three main types of experiments: survey experiments, lab experiments (including lab-in-the-field experiments), and field experiments.

If you want to do a survey experiment, the simplest option is Qualtrics, a software program for which the College maintains a subscription for both computer-based and mobile platforms (for experiments you want to do remotely). Qualtrics can be programmed in very sophisticated ways to randomly assign subjects to different treatments.

One of the drawbacks of survey experiments that subjects take in the comfort of their own environment is a lack of control on the part of the researcher in controlling that environment. For example, if you are conducting your survey on an Internet sample, subjects are able to browse the Internet or walk away from the computer while they are taking your study. This is most problematic if there is reason to think that some facet of your treatment might make subjects more likely to get distracted or visit other webpages, for example if your treatment is very long or if you ask post-test questions about political knowledge, where subjects might feel inclined to seek out the correct answers online. Programs like Qualtrics have some built in functionalities to be able to detect this.

However, in some instances, you may want to conduct a survey experiment in the laboratory, either to exert more control over the experiment or because you want to deliver a treatment that is not well suited to the online or phone format. In that instance you may want to use the research lab facilities of the SSRMC. If you are doing a lab experiment and you are interested in delivering media (images, audio, or videos) to subjects in a laboratory environment, one of the most popular software programs to do so is SuperLab. The SSRMC currently has licenses for this program on one computer. An example honors thesis that relies on SuperLab for stimulus delivery can be found here.

If you are interested in doing an experiment in the field, academics frequently partner with outside organizations—such as campaigns or advocacy organizations—because it is often difficult to get access to the large subject pools necessary to conduct these studies. William & Mary undergraduates have successfully done this, and this honors thesis is a great example.

3. Analyzing the Data

There are more sophisticated ways of analyzing the data from experiments, and methodologists are constantly developing new ways to extract less biased estimates of the causal effects in a study. While these more advanced approaches are beyond the scope of this introductory module, the resources listed at the end of this module contain more in-depth information.

4. Considerations and Cautions

banner-1165976_1920

Studies involving human subjects must go through ethics review for good reason, and this is especially important in the case of experiments, where researchers manipulate the environment or stimuli to which participants are exposed.

Different experimental traditions in the social sciences have different norms. One of the biggest differences is in the instructions that researchers give participants about the nature of the experimental tasks. Experiments rooted in psychology often allow researchers to use mild deception in their instructions if the researcher thinks that knowing the true purpose of the study would alter the way that participants behave. The SSRMC allows deception in studies, as long as that deception is approved by the Institutional Review Board. However, deception is almost always avoided in experiments rooted in economics. A second major difference between economic and psychology experiments is whether (or how) subjects are incentivized for their participation.

The classic tradeoff in experimental design is between internal validity and external validity. While experiments have high internal validity, to varying degrees, they may lack external validity, the ability of a researcher to make claims about how the results of the study would generalize and hold up in different contexts. One particularly common generalizability concern stems from differences in the sample used in your experiment compared the population to which you want to generalize. When is it problematic to generalize the findings of an experiment conducted on a convenience (often, student) population? First, it is important to know how the student population differs from a more representative population. The obvious answers are in age, education level, and geographic location.

But other factors can matter as well. The key question to ask is “how else are college students different in a way that should affect the strength or direction of the causal relationship I am testing?” If your convenience sample is different in a way that makes it harder to find the relationship you observe, then you can assert that your study likely underestimates the relationship between the variables in a more representative population (a testable proposition!) However, if your sample makes it easier to find effects, then generalizability concerns become more serious. Sometimes, these concerns are very large. For example, college students are particularly susceptible to conformity (Sears, 1986), which could be important depending on the nature of your study. The Omnibus Project draws from courses in the government and international relations program, suggesting that our participants have a greater interest and level of knowledge about politics. These factors may help or hurt your ability to make claims about how general your findings are.

More broadly speaking, researchers have commented on the abundance of WEIRD subjects in experimental studies: subjects that are Western, educated and come from countries that are industrialized, rich and democratic. This has been written about considerably in the popular media and has been addressed in academic research as well.

5. Additional Resources

There are many excellent resources available online, through SWEM library, and through the SSRMC research methods collection. The module below draws on information from these sources, but there is much more detail available in the original sources.

Experimental Design

  • Druckman, James N. et al. 2011. Cambridge Handbook of Experimental Political Science.
  • Rebecca Morton and Kenneth Williams. 2010. From Nature to the Lab: Experimental Political Science and the Study of Causality
  • McDermott, Rose. 2002. “Experimental Methods in Political Science.” Annual Review of Political Science 5:31-61.
  • McDermott, Rose. 2002. “Experimental Methodology in Political Science.” Political Analysis 10(4):325-342
  • Druckman, James, Donald Green, James Kuklinkski, and Arthur Lupia. 2006. “The Growth and Development of Experimental Research in Political Science,” American Political Science Review 100:627-635
  • Gaines, Brian et al. 2006. “The Logic of the Survey Experiment Reexamined.” Political Analysis
  • Imai,Kosuke et al. 2011. “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review

Field Experiments

  • Green and Gerber’s Field Experiments book
  • Yales ISPS Field Experiments Initiative as well as their Data Archive