The field of psychometrics is all about this kind of thing. The keyword here is “practice effect”. It sort of looks like Wikipedia’s deletionists have trimmed their content on the concept down to two sentences in an article that’s been nominated for deletion as non-notable, but if you hunt around you can find pre-existing content on ways to control for practice effects.
The unique thing about the situation with LW in this respect seems to me to be that there are a lot of people who tend to conceive and execute and publish polls with relatively sophisticated methodology for the internet in the complete absence of grants or formal publication or whatever. We’re doing as a hobby for a community blog, (yes, a blog) what academics make an entire career out of! Maybe not fully solid with control groups yet, but we’re kind of close to this already actually.
To make this sort of “surprising but casual competence” more dramatic and effective, it seems like it might be worthwhile to do a sequence on current best practices for community members to run studies on the community itself. Between the free polling technology available via the forms built into google docs and and chapters from psychometrics textbooks, I bet it wouldn’t be that hard for someone to pull together such content for a sequence that makes it easier for people here to spice their efforts up with pretty solid techniques :-)
For example, we probably could get some interesting control data for practice effects the way Anna got control data in her recent poll via mechanical turk. If you’ve developed the quiz content you could ask LWers to take it and turkers to take it, and then get the same LWers and turkers to re-take it with some being exposed to whatever manipulation you tried on LW by posting content and others not… it wouldn’t be a perfect control, but (ignoring the costs) it would be better than nothing...
...which naturally leads me to wonder. What is the value of information here? Is there some change in behavior that certain results would cause? What kind of increase in value could be expected from such a change in behavior? Anyone have guesses here?
What is the value of information here? Is there some change in behavior that certain results would cause? What kind of increase in value could be expected from such a change in behavior? Anyone have guesses here?
If the sequence was shown to be useful we would be able to use the data to help show people that LW is useful. If the sequence is not useful we would likely need to do more research to determine why. If we find that the sequence was merely ineffective in instilling the techniques we could re-write it to be more effective. If it turns out that the techniques themselves are ineffective we could stop teaching them. Preferably we wouldn’t remove the sequence just add a warning at the start of each post. This would save people time and encourage them to create alternative techniques.
The field of psychometrics is all about this kind of thing. The keyword here is “practice effect”. It sort of looks like Wikipedia’s deletionists have trimmed their content on the concept down to two sentences in an article that’s been nominated for deletion as non-notable, but if you hunt around you can find pre-existing content on ways to control for practice effects.
The unique thing about the situation with LW in this respect seems to me to be that there are a lot of people who tend to conceive and execute and publish polls with relatively sophisticated methodology for the internet in the complete absence of grants or formal publication or whatever. We’re doing as a hobby for a community blog, (yes, a blog) what academics make an entire career out of! Maybe not fully solid with control groups yet, but we’re kind of close to this already actually.
To make this sort of “surprising but casual competence” more dramatic and effective, it seems like it might be worthwhile to do a sequence on current best practices for community members to run studies on the community itself. Between the free polling technology available via the forms built into google docs and and chapters from psychometrics textbooks, I bet it wouldn’t be that hard for someone to pull together such content for a sequence that makes it easier for people here to spice their efforts up with pretty solid techniques :-)
For example, we probably could get some interesting control data for practice effects the way Anna got control data in her recent poll via mechanical turk. If you’ve developed the quiz content you could ask LWers to take it and turkers to take it, and then get the same LWers and turkers to re-take it with some being exposed to whatever manipulation you tried on LW by posting content and others not… it wouldn’t be a perfect control, but (ignoring the costs) it would be better than nothing...
...which naturally leads me to wonder. What is the value of information here? Is there some change in behavior that certain results would cause? What kind of increase in value could be expected from such a change in behavior? Anyone have guesses here?
If the sequence was shown to be useful we would be able to use the data to help show people that LW is useful. If the sequence is not useful we would likely need to do more research to determine why. If we find that the sequence was merely ineffective in instilling the techniques we could re-write it to be more effective. If it turns out that the techniques themselves are ineffective we could stop teaching them. Preferably we wouldn’t remove the sequence just add a warning at the start of each post. This would save people time and encourage them to create alternative techniques.