Great idea. One potential problem though for these sorts of experiments is that knowledge (or reasonable suspicion) of the experiments would alter users’ behavior.
Yes, but I’m hoping using a randomly selected minority posts or comments would help, and I’d expect our estimations as to which posts have been raised or lowered would be interestingly inaccurate. Maybe we could submit our guesses along with the probability we assign to each guess, then the calibration test results could be posted… :-)
Great idea. One potential problem though for these sorts of experiments is that knowledge (or reasonable suspicion) of the experiments would alter users’ behavior.
Yes, but I’m hoping using a randomly selected minority posts or comments would help, and I’d expect our estimations as to which posts have been raised or lowered would be interestingly inaccurate. Maybe we could submit our guesses along with the probability we assign to each guess, then the calibration test results could be posted… :-)