Thanks for your feedback. While I agree with you at many (even most) points, there are several considerations to keep in mind:
It is not possible for me to change the questions at this point in time, There are nearly 200 responses at this point and it would be completely unfair of me to force everyone to retake the survey so I can fix most of the offered feedback. As a consequence I can only take these as potential improvements for the next survey.
The questions in the AI Progress section are ripped directly from an associated study, I have no control over their content or methodology besides replicating them as accurately and faithfully as I can.
Some simplification of scenarios is necessary to make them fit into a <150 question survey. There are also real limits to how much effort I can expect from people in terms of engaging with a scenario and that is why certain things such as the probability that a genetic treatment will be safe aren’t included. Many things could use a probability but if I asked for one every time people would probably get frustrated with the survey and give up.
In the future we’re looking at changing platforms so that the survey can be offered in a ‘module’ format which allows it to be taken in chunks over a much longer period of time with more detail in individual sections.
Quite often it is useful for a survey to explicitly not include a neutral option. Picking between two things is mentally difficult and a neutral option offers a path of least resistance which people are more likely to click than put in the effort even if that doesn’t represent their true opinion.
At analysis time I have access to a respondents previous answers.
Our survey software prevents me from doing certain things which would be desirable.
Responding to certain specific questions:
Q33: Yes, CafeChesscourt. The question is “given nothing but the appearance of this forum” (CafeChesscourt put approximately zero effort into software, making it a useful control about how important software is to a forums success), would you use it if one of these “celebrity users” listed below were someone that ran it and used it regularly? The purpose of this question is to gauge how useful it is to get people who are popular to endorse a discussion forum. And while you might think it’s bonkers to choose a forum that way, empirically many people will show up to places that are often reviled like Tumblr if the right person asks them to.
Q33ff: I agree that is an assumption many people are making in regards to this issue, that assumption is not there in the question however as its purpose was me teasing at the idea that perhaps focusing on technical excellence is the wrong metric.
Q36: I believe this is a limitation of the software, though I’ll go ahead and double check since letting people write stuff in wouldn’t materially effect the survey results.
Q52: Depression does go away more often than the others. But more importantly on the 2016 survey we had somewhat horrifying rates of depression. It was difficult to distinguish if this was because everybody gets depression at least once and then ‘gets over it’ or because that many LessWrongers are actually depressed. This question lets us tell the difference.
Q67: Yes. As you can tell this is a hard question to ask people but I think it’s important enough to be worth asking even if imperfectly. In a future survey Calculus could probably be changed to “Calculus or ‘higher maths’”. The painting/etc option is meant to apply to visual arts. In the future it would probably be better to more accurately specify. In the grand scheme of things it will probably not spoil this years survey results to have the occasional musician pick it under the impression it counts. (I should also add music, thanks for the tip.)
Update (Fri Sep 15 13:46:59 PDT 2017): The issue in question 36 turned out to be fixable, thanks for the help!
For the avoidance of doubt, I appreciate your “several considerations” and in particular was not suggesting that you should hack the questions about mid-survey. And, er, I realise that I just posted a bunch of criticism without adding: thank you very much for doing the survey; I think it will be interesting and useful; the fact that I have some quibbles doesn’t make that any less true. So please consider that added :-).
As a single data point, on Q33 I attempted to answer as if the question meant something like “If all you knew about a new forum was that X was running the show, would you be likely to check it out for that reason?” on the grounds that that was the most non-bonkers interpretation I could give the question. If it was meant to be more like “If X was running the show, and the forum had no other merits, would X’s leadership be enough to make you use it and stick around despite its lack of other merits?” then my answer, at least, will not be informative. I suspect I’m not alone :-).
Thanks for your feedback. While I agree with you at many (even most) points, there are several considerations to keep in mind:
It is not possible for me to change the questions at this point in time, There are nearly 200 responses at this point and it would be completely unfair of me to force everyone to retake the survey so I can fix most of the offered feedback. As a consequence I can only take these as potential improvements for the next survey.
The questions in the AI Progress section are ripped directly from an associated study, I have no control over their content or methodology besides replicating them as accurately and faithfully as I can.
Some simplification of scenarios is necessary to make them fit into a <150 question survey. There are also real limits to how much effort I can expect from people in terms of engaging with a scenario and that is why certain things such as the probability that a genetic treatment will be safe aren’t included. Many things could use a probability but if I asked for one every time people would probably get frustrated with the survey and give up.
In the future we’re looking at changing platforms so that the survey can be offered in a ‘module’ format which allows it to be taken in chunks over a much longer period of time with more detail in individual sections.
Quite often it is useful for a survey to explicitly not include a neutral option. Picking between two things is mentally difficult and a neutral option offers a path of least resistance which people are more likely to click than put in the effort even if that doesn’t represent their true opinion.
At analysis time I have access to a respondents previous answers.
Our survey software prevents me from doing certain things which would be desirable.
Responding to certain specific questions:
Q33: Yes, CafeChesscourt. The question is “given nothing but the appearance of this forum” (CafeChesscourt put approximately zero effort into software, making it a useful control about how important software is to a forums success), would you use it if one of these “celebrity users” listed below were someone that ran it and used it regularly? The purpose of this question is to gauge how useful it is to get people who are popular to endorse a discussion forum. And while you might think it’s bonkers to choose a forum that way, empirically many people will show up to places that are often reviled like Tumblr if the right person asks them to.
Q33ff: I agree that is an assumption many people are making in regards to this issue, that assumption is not there in the question however as its purpose was me teasing at the idea that perhaps focusing on technical excellence is the wrong metric.
Q36: I believe this is a limitation of the software, though I’ll go ahead and double check since letting people write stuff in wouldn’t materially effect the survey results.
Q52: Depression does go away more often than the others. But more importantly on the 2016 survey we had somewhat horrifying rates of depression. It was difficult to distinguish if this was because everybody gets depression at least once and then ‘gets over it’ or because that many LessWrongers are actually depressed. This question lets us tell the difference.
Q67: Yes. As you can tell this is a hard question to ask people but I think it’s important enough to be worth asking even if imperfectly. In a future survey Calculus could probably be changed to “Calculus or ‘higher maths’”. The painting/etc option is meant to apply to visual arts. In the future it would probably be better to more accurately specify. In the grand scheme of things it will probably not spoil this years survey results to have the occasional musician pick it under the impression it counts. (I should also add music, thanks for the tip.)
Update (Fri Sep 15 13:46:59 PDT 2017): The issue in question 36 turned out to be fixable, thanks for the help!
For the avoidance of doubt, I appreciate your “several considerations” and in particular was not suggesting that you should hack the questions about mid-survey. And, er, I realise that I just posted a bunch of criticism without adding: thank you very much for doing the survey; I think it will be interesting and useful; the fact that I have some quibbles doesn’t make that any less true. So please consider that added :-).
As a single data point, on Q33 I attempted to answer as if the question meant something like “If all you knew about a new forum was that X was running the show, would you be likely to check it out for that reason?” on the grounds that that was the most non-bonkers interpretation I could give the question. If it was meant to be more like “If X was running the show, and the forum had no other merits, would X’s leadership be enough to make you use it and stick around despite its lack of other merits?” then my answer, at least, will not be informative. I suspect I’m not alone :-).