I think it’s possible to exercise Hufflepuff virtue in the act of encouraging more Ravenclaw virtue, right? That is, getting an arbitrary ball rolling is a Hufflepuff thing to do, even if you roll the ball in a Ravenclaw direction? That’s an important distinction to me.
A mid-term goal of mine is to replicate Dougherty et al.‘s MINERVA-DM in MIT/GNU Scheme (it was originally written in Pascal; no, I haven’t requested the authors’ source code, and I don’t intend to). I also intend to test at least one of its untested predictions using Amazon Mechanical Turk, barring any future knowledge that makes me think that I won’t be able to obtain reliable results (which has only become less plausible as I’ve learned more; e.g. Turkers are more representative of the U.S. population than the undergraduate population that researchers routinely sample from in behavioral experiments; there’s also a few enthusiasts who have done some work on AMT-specific methodological considerations).
It’s worth saying that I’m not very confident that MINERVA-DM won’t be overturned by a better model, and that’s not the point.
I need some sort of example, and MINERVA-DM has good properties as an example, because its math is exceedingly simple (i.e., capital-sigma notation, arithmetic mean, basic probability theory (see Bolstad’s Introduction to Bayesian Statistics, Ch.3), etc. There are probably plenty of improvements that we need to and could make as a community, but my own concern is that it’s never been winter-night-clear to me why at least some of us aren’t trying to perform (Keyword Alert!) heuristics and biases/judgment and decision making (JDM)/behavioral decision theory research on LW or on whatever conversational focus we may be using in the near- to mid-term future. There is no organization in the community for this; CFAR is the closest thing to this, and AFAICT, they are not doing basic research into H&B/JDM/BDT. People around here seem to me more likely than most to agree that you’re more likely to make progress on applications if you have a deep understanding of the problem that you’re trying to solve.
I think it is intuitive that you simply cannot productively do academic work solely in the blogosphere, and when you’re explaining a counterintuitive point, a point that is not universal background knowledge, you should recurse far enough to prevent misunderstandings in advance. I no longer find it intuitive that you can’t do a substantial amount of work on the blogosphere. For one, a good deal of academic work, especially the kind we’re collectively interested in, doesn’t require any special resources. Reviews, syntheses, analyses, critiques, and computational studies can all be done from a keyboard. As for experiments, we don’t need to buy a particle accelerator for psych research, you guys; this is where Mechanical Turk comes in. E.g. see these two blogposts wherein a scientist replicates one of Tversky and Kahneman’s base rate fallacy results with N = 66 for US$3.30, and replicates one of Tversky and Kahneman’s conjunction fallacy results with N = 50 for US$2.50. (Here’s a list with more examples.)
Arguing that there’s important academic work that doesn’t require anything but a computer (reviews, syntheses, analyses, computational studies), and demonstrating that you can test experimental predictions with your lunch money seems like a good start on preempting the ‘you can’t do real science outside of academia’ criticism. (It’s not like there isn’t a precedent for this sort of thing around here anyway.) It also prevents people from calling you a hypocrite for proposing that the community steer in a certain direction without your doing any of the pedaling. I probably would’ve kept quiet for a lot longer if I didn’t think it were important to the community to respond to calls like this article, especially considering that we may be moving to a new platform soon.
I think it’s possible to exercise Hufflepuff virtue in the act of encouraging more Ravenclaw virtue, right? That is, getting an arbitrary ball rolling is a Hufflepuff thing to do, even if you roll the ball in a Ravenclaw direction? That’s an important distinction to me.
A mid-term goal of mine is to replicate Dougherty et al.‘s MINERVA-DM in MIT/GNU Scheme (it was originally written in Pascal; no, I haven’t requested the authors’ source code, and I don’t intend to). I also intend to test at least one of its untested predictions using Amazon Mechanical Turk, barring any future knowledge that makes me think that I won’t be able to obtain reliable results (which has only become less plausible as I’ve learned more; e.g. Turkers are more representative of the U.S. population than the undergraduate population that researchers routinely sample from in behavioral experiments; there’s also a few enthusiasts who have done some work on AMT-specific methodological considerations).
MINERVA-DM is a formal model of human likelihood judgments that successfully predicts the experimental findings on conservatism), the availability heuristic, the representativeness heuristic, the base rate fallacy, the conjunction fallacy, the illusory truth effect, the simulation heuristic, and the hindsight bias. MINERVA-DM can also be described as a modified version of Bayes’ Theorem. I’m not too far yet, having just started learning Scheme/programming-in-general, but I have managed to hobble together a one-line program that outputs an n-vector with elements drawn randomly with replacement from the set {-1, 0, 1}, so I guess I’ve technically started writing the program.
It’s worth saying that I’m not very confident that MINERVA-DM won’t be overturned by a better model, and that’s not the point.
I need some sort of example, and MINERVA-DM has good properties as an example, because its math is exceedingly simple (i.e., capital-sigma notation, arithmetic mean, basic probability theory (see Bolstad’s Introduction to Bayesian Statistics, Ch.3), etc. There are probably plenty of improvements that we need to and could make as a community, but my own concern is that it’s never been winter-night-clear to me why at least some of us aren’t trying to perform (Keyword Alert!) heuristics and biases/judgment and decision making (JDM)/behavioral decision theory research on LW or on whatever conversational focus we may be using in the near- to mid-term future. There is no organization in the community for this; CFAR is the closest thing to this, and AFAICT, they are not doing basic research into H&B/JDM/BDT. People around here seem to me more likely than most to agree that you’re more likely to make progress on applications if you have a deep understanding of the problem that you’re trying to solve.
I think it is intuitive that you simply cannot productively do academic work solely in the blogosphere, and when you’re explaining a counterintuitive point, a point that is not universal background knowledge, you should recurse far enough to prevent misunderstandings in advance. I no longer find it intuitive that you can’t do a substantial amount of work on the blogosphere. For one, a good deal of academic work, especially the kind we’re collectively interested in, doesn’t require any special resources. Reviews, syntheses, analyses, critiques, and computational studies can all be done from a keyboard. As for experiments, we don’t need to buy a particle accelerator for psych research, you guys; this is where Mechanical Turk comes in. E.g. see these two blog posts wherein a scientist replicates one of Tversky and Kahneman’s base rate fallacy results with N = 66 for US$3.30, and replicates one of Tversky and Kahneman’s conjunction fallacy results with N = 50 for US$2.50. (Here’s a list with more examples.)
Arguing that there’s important academic work that doesn’t require anything but a computer (reviews, syntheses, analyses, computational studies), and demonstrating that you can test experimental predictions with your lunch money seems like a good start on preempting the ‘you can’t do real science outside of academia’ criticism. (It’s not like there isn’t a precedent for this sort of thing around here anyway.) It also prevents people from calling you a hypocrite for proposing that the community steer in a certain direction without your doing any of the pedaling. I probably would’ve kept quiet for a lot longer if I didn’t think it were important to the community to respond to calls like this article, especially considering that we may be moving to a new platform soon.