Mturk: An alternative to the college sample?

October 14, 2014 4:56 PM | Shannon Claxton

Research on sexuality during emerging adulthood is often characterized by samples of college students. Although in many cases these samples are appropriate (college is an important aspect of the lives of many emerging adults), reliance on college samples can also be problematic. The use of college samples poses potential problems for generalizability. Many emerging adults do not attend college, and the college environment has a number of unique aspects that make it different from the “the real world.” I’m certainly not the first to mention the potential issues with using convenience samples of college students, and I’m not going to debate the limitations here (see Henrich, Heine, & Norenzayan, 2010; and Peterson, 2001 for discussions).

My purpose, rather, is to discuss one of the alternative techniques researchers are using to recruit from different populations. In particular, a number of researchers have turned to Amazon’s Mechanical Turk (Mturk) and other online data collection sources to obtain data from non-college samples of emerging adults. I have collected data from Mturk for a number of research projects, and it can be wonderful – instead of taking months or even years to collect data, it is possible to collect data from hundreds of individuals in a few days. Furthermore, it’s relatively inexpensive to collect data on Mturk (average payments for surveys run less than $1 - Buhrmester, Kwang, & Gosling, 2011).

Although the ease of online data collection makes it appealing, one may question the representativeness of these samples. Are emerging adults who take surveys for minimal pay different from other emerging adults? Research so far suggests that these samples are more representative of the U.S. population than typical college samples (e.g., Paolacci, Chandler & Ipeirotis, 2010 ) and that results obtained on Mturk are similar to results from in-person studies (e.g., Casler, Bickel, & Hackett, 2013). However, collecting data without ever seeing participants can be disconcerting. Are these participants really who they say they are? How can researchers be sure that participants are answering truthfully? While validity is important for any type of data collection, one wonders if anonymous, online data collection methods such as Mturk lead participants to be more honest about sensitive topics such as sexuality or if the feeling of anonymity creates additional opportunities for lying/misrepresentation.

What do you think? – Is online data collection a good way to tap into a more diverse population? What other options are out there? What techniques have you used – and were they effective?  What have you done in your research to make sure that data from online sources is reliable? Do you see additional pros/cons of these data collection methods? Share your experiences in the comments!