Great article, but as a sidenote: It seems that the spike experiment would be worthwhile running even if it isn’t the only theory that could predict the spike. After all, if no current theory predicts the spike, they may very well need to be modified or even abandoned in light of the experiment. Further, challenging theories that are popular seems more important than challenging theories that no-one believes yet.
I agree, you still probably run the experiment. At least it codifies something which was previously implicit or “folk-knowledge” among the scientists. G-complex theory is a (possibly wrong) precise statement of existing intuitions. I would now say that Dr. Y has made a gears-level explanation where one did not previously exist.
If all we care about is gears-level, then assigning high credence to Dr. Y’s explanation after the experiment succeeds is a good thing. This is largely true in science; the theory deserves the credit for the explicit prediction.
However, if all we care about is accuracy, something has gone wrong. People are likely to assign too much credence to G-complex theory. Everyone familiar with the area already knew how the experiment would turn out, without G-complex theory.
A possible remedy is to gather predictions from other scientists before performing the experiments. The other scientists might report low endorsement of G-complex theory, but expect the spike.
If everyone expects a spike and it doesn’t happen, you now know that it’s really a pretty big deal.
Great article, but as a sidenote: It seems that the spike experiment would be worthwhile running even if it isn’t the only theory that could predict the spike. After all, if no current theory predicts the spike, they may very well need to be modified or even abandoned in light of the experiment. Further, challenging theories that are popular seems more important than challenging theories that no-one believes yet.
I agree, you still probably run the experiment. At least it codifies something which was previously implicit or “folk-knowledge” among the scientists. G-complex theory is a (possibly wrong) precise statement of existing intuitions. I would now say that Dr. Y has made a gears-level explanation where one did not previously exist.
If all we care about is gears-level, then assigning high credence to Dr. Y’s explanation after the experiment succeeds is a good thing. This is largely true in science; the theory deserves the credit for the explicit prediction.
However, if all we care about is accuracy, something has gone wrong. People are likely to assign too much credence to G-complex theory. Everyone familiar with the area already knew how the experiment would turn out, without G-complex theory.
A possible remedy is to gather predictions from other scientists before performing the experiments. The other scientists might report low endorsement of G-complex theory, but expect the spike.
If everyone expects a spike and it doesn’t happen, you now know that it’s really a pretty big deal.