Episode 99: Is MTurk Too Good To Be True?

Two Psychologists Four Beers

In a recent article, psychologists Webb and Tangney document their experience collecting psychology data online using Amazon's crowdsourcing platform MTurk. Alarmingly, the authors conclude that ultimately only 2.6% of their sample was valid data from human beings. Yoel and Alexa weigh in on these findings, discussing what researchers can reasonably expect from online studies and platforms, and how their personal experiences have informed their own practices. They also consider a response written by Cuskley and Sulik, who argue that researchers, not recruitment platforms, are responsible for ensuring the quality of data collected online. Questions that arise include: What studies do people want to do? Does anyone read the fine print? And what are the ethics of mouse-hunting?

Links:

• Two Psychologists Four Beers on Untappd
• Too Good to Be True: Bots and Bad Data From Mechanical Turk - Margaret A. Webb, June P. Tangney, 2022
• PsyArXiv Preprints | The burden for high-quality online data collection lies with researchers, not recruitment platforms
• Living in Harmony with House Mice and Rats | PETA