Unless, of course, your concern is that the subjects are lying about their demographics
Yes. Or, rather, the subjects submit noise as data.
Consider, e.g. a Vietnamese teenager who knows some English and has declared himself as an American to MTurk. He’ll fill out a survey because he’ll get paid for it, but there is zero incentive for him to give true answers (and some questions like “Did you vote for Obama?” are meaningless for him). The rational thing for him to do is to put checkmarks into boxes as quickly as he can without being obvious about his answers being random.
Instead of dismissing MTurk based on expectations that it would be useless for research, I think it would be important to test it.
I’ll rephrase this as “it would be useful and necessary to test it before we use MTurk samples for research”.
The rational thing for him to do is to put checkmarks into boxes as quickly as he can without being obvious about his answers being random.
This is a good point. You still would be able to match the resulting demographics to known trends and see how reliable your sample is, however. Random answers should show, either overtly on checks, or subtlety through aggregate statistics.
~
I’ll rephrase this as “it would be useful and necessary to test it before we use MTurk samples for research”.
The paper I mean is that one: http://cpx.sagepub.com/content/early/2013/01/31/2167702612469015.abstract
Yes. Or, rather, the subjects submit noise as data.
Consider, e.g. a Vietnamese teenager who knows some English and has declared himself as an American to MTurk. He’ll fill out a survey because he’ll get paid for it, but there is zero incentive for him to give true answers (and some questions like “Did you vote for Obama?” are meaningless for him). The rational thing for him to do is to put checkmarks into boxes as quickly as he can without being obvious about his answers being random.
I’ll rephrase this as “it would be useful and necessary to test it before we use MTurk samples for research”.
Here you go.
~
This is a good point. You still would be able to match the resulting demographics to known trends and see how reliable your sample is, however. Random answers should show, either overtly on checks, or subtlety through aggregate statistics.
~
Definitely.