Thank you for the references! Aella’s case is interesting a very good example ofhow design matters to how you should think about the problem, particularly her statement (emphasis added):
in general whenever I’ve asked variations on this set of questions, the number of correct votes roughly increases with the difficulty of the problem, before hitting a ceiling when the difficulty is too high, and slowly decreases with that difficulty.
In a lot of survey design the opposite is a much larger problem, particularly if you incentivize responses (i.e. paying people for completing the survey). The more complex you make a question, the more often you will have lazy respondents just pick the first question instead of engaging with what is being asked. Because she is just posing a twitter poll with little incentive to answer, I imagine most lazy respondents simply do not read the question.
Thank you for the references! Aella’s case is interesting a very good example of how design matters to how you should think about the problem, particularly her statement (emphasis added):
In a lot of survey design the opposite is a much larger problem, particularly if you incentivize responses (i.e. paying people for completing the survey). The more complex you make a question, the more often you will have lazy respondents just pick the first question instead of engaging with what is being asked. Because she is just posing a twitter poll with little incentive to answer, I imagine most lazy respondents simply do not read the question.