“Now, just to be clear,” Harry said, “if the professor does levitate you, Dad, when you know you haven’t been attached to any wires, that’s going to be sufficient evidence. You’re not going to turn around and say that it’s a magician’s trick. That wouldn’t be fair play. If you feel that way, you should say so now, and we can figure out a different experiment instead.”
Harry’s father, Professor Michael Verres-Evans, rolled his eyes. “Yes, Harry.”
“And you, Mum, your theory says that the professor should be able to do this, and if that doesn’t happen, you’ll admit you’re mistaken. Nothing about how magic doesn’t work when people are sceptical of it, or anything like that.”
An important aspect of a scientific experiment is that you figure out the design of your experiment and how you are going to interpret the resulting data before you execute it. When you are using an experiment to resolve a disagreement, everyone involved should agree to this procedure in advance (assuming trust in intellectual honesty, which I think holds here).
Polls and surveys have self selection issues. It is good to take some step to counteract the tendency of blog readers to like reading stuff on blogs, but is it sufficient? Should Luke be convinced the bias has been remedied?
Luke was arguing in part that academic papers published in journals helps to reach a small but highly valued class of people. Should a poll that doesn’t track his value of reaching the participants influence his policy decision?
I think it would be a better process, if Tom first presented the design of the poll, and allow some time for the community to critique the design. Only after modifications have been made to address criticisms, and Luke and Tom agree that it is a valid test of the question they are interested in, should the poll itself (if that is still the form of the experiment) be conducted.
I expect Luke has put a lot of hours into figuring out what formats to publish information in, including gathering information about the preferences of people he wants to reach. Do you expect to do better yourself in less time?
-- HPMOR, chapter 2
An important aspect of a scientific experiment is that you figure out the design of your experiment and how you are going to interpret the resulting data before you execute it. When you are using an experiment to resolve a disagreement, everyone involved should agree to this procedure in advance (assuming trust in intellectual honesty, which I think holds here).
Polls and surveys have self selection issues. It is good to take some step to counteract the tendency of blog readers to like reading stuff on blogs, but is it sufficient? Should Luke be convinced the bias has been remedied?
Luke was arguing in part that academic papers published in journals helps to reach a small but highly valued class of people. Should a poll that doesn’t track his value of reaching the participants influence his policy decision?
I think it would be a better process, if Tom first presented the design of the poll, and allow some time for the community to critique the design. Only after modifications have been made to address criticisms, and Luke and Tom agree that it is a valid test of the question they are interested in, should the poll itself (if that is still the form of the experiment) be conducted.
I would agree if I were going to spend a lot of hours on this, but I unfortunately don’t have that kind of time.
Still, I’d like to see more “measure twice, cut once” here.
If you measure once, you might have to cut twice, or you might just have to throw it away.
I expect Luke has put a lot of hours into figuring out what formats to publish information in, including gathering information about the preferences of people he wants to reach. Do you expect to do better yourself in less time?