Do Results Generalize for Privacy and Security Surveys

Platform Insights

Jenny Tang, a Ph.D. student at Carnegie Mellon University, explores the generalizability of survey platforms like MTurk and Prolific. She highlights that while MTurk was once considered reliable for certain demographics, recent findings suggest a decline in data quality, possibly due to bot activity 1. Prolific, on the other hand, tends to provide better data quality, especially for security and privacy surveys 2. Jenny notes, "Anecdotally, it seems that our data is not as good as it used to be, and so what can we do?" 1.

   

Attention Checks

Attention checks are crucial for ensuring data quality in online surveys. Jenny discusses various techniques, such as instruction manipulation checks and text box attention checks, which are more effective than captchas in identifying non-human responses 3. She emphasizes the importance of choosing the right type of attention check based on the platform used, suggesting text box checks for MTurk due to its susceptibility to bot responses 4. Jenny explains, "If you believe there are bots, use text box attention checks over other types of attention checks" 4.

   

Demographic Analysis

Jenny examines the demographic representation of survey samples from MTurk and Prolific, noting significant differences from the general population. Prolific samples tend to be more representative, especially with options for gender balance and representative sampling 5. However, both platforms skew towards younger, more educated users, which may not reflect the broader population 6. Jenny remarks, "Our sample population often isn't [the target population]... people who are on these platforms, who are willing to take surveys" 7.