It is possible there simply isn’t any such experimental material. If I had to bet on it I would say it is more likely there is some than not, though I would also bet that some things we wish where done haven’t been so far. In the past I’ve wondered if we can in the future expect CFAR or LessWrong to do experimental work to test many of the hypotheses based on insight or long fragile chains of reasoning we’ve come up with. I don’t think I’ve seen anyone talk about considering this.
While mention of say CFAR doing this, the mind jumps to them doing expensive experiments or posing long questionnaires with small samples of psychology students and then publishing papers, like everyone else does. This is something that may not be worth their effort but seems doable, the idea of LWers getting into the habit of testing their ideas on human rationality however seems utterly impractical.
Or is it? How useful would it be if we had something like this site visited by thousands or tens of thousands, where high karma LessWrong posters and CFAR researchers could submit their quizzes and online experiments. How useful would it be if we made such a data set publicly available? What if we could in addition to this data mine how people use apps or an online rationality class?
Would it be useful? I think it would. Would it be used? There are many publicly available data sets and we see little if any original analysis based on them here. We either don’t have norms encouraging this or we don’t have enough people comfortable with statistics doing so. Problems like this aren’t immutable. The Neglected Virtue of Scholarship noticeably changed our community in a similarly profound way with positive results. Is building knowledge this way even possible in a field that takes years to study? A fair question, especially for tasks that require technical competence, the answer is yes.
It is possible there simply isn’t any such experimental material. If I had to bet on it I would say it is more likely there is some than not, though I would also bet that some things we wish where done haven’t been so far. In the past I’ve wondered if we can in the future expect CFAR or LessWrong to do experimental work to test many of the hypotheses based on insight or long fragile chains of reasoning we’ve come up with. I don’t think I’ve seen anyone talk about considering this.
While mention of say CFAR doing this, the mind jumps to them doing expensive experiments or posing long questionnaires with small samples of psychology students and then publishing papers, like everyone else does. This is something that may not be worth their effort but seems doable, the idea of LWers getting into the habit of testing their ideas on human rationality however seems utterly impractical.
Or is it? How useful would it be if we had something like this site visited by thousands or tens of thousands, where high karma LessWrong posters and CFAR researchers could submit their quizzes and online experiments. How useful would it be if we made such a data set publicly available? What if we could in addition to this data mine how people use apps or an online rationality class?
Would it be useful? I think it would. Would it be used? There are many publicly available data sets and we see little if any original analysis based on them here. We either don’t have norms encouraging this or we don’t have enough people comfortable with statistics doing so. Problems like this aren’t immutable. The Neglected Virtue of Scholarship noticeably changed our community in a similarly profound way with positive results. Is building knowledge this way even possible in a field that takes years to study? A fair question, especially for tasks that require technical competence, the answer is yes.