I didn’t understand Psychohistorian’s post as suggesting that we should make up fictional data—for then of course it may be no surprise that the given theory would have to bend in order to accommodate it. Rather, we should take real data, which is not explained by the theory (but which is understood in light of some different theory), and see just how easily the advocate can stretch his explanation to accommodate it. Does he/she notice the stretch? Can he/she resolve the difference between that data from the others?
What should be the take away lesson from this?
People get into man-with-a-hammer mode with evolutionary explanations. A lot. Because of the nature of evolutionary biology, sometimes they just reason like, “I can imagine what advantages this feature could have conferred in the past. Thus, …”. And yes, a lot of the time what you get is ad hoc crap.
But what if we don’t know which data is actually explained by the theory or not? That will make it hard to come up with “real data, which is not explained by the theory”.
Not quite. The idea is to see if the theory can convincingly explain fake data. If it can, it doesn’t mean the theory is wrong, it just means your capacity to infer things from it is limited. Natural selection is interesting and useful, but it is not a reliable predictor in many cases. You routinely see people say, “The market must do X because of Y.” If they could say basically the exact same thing about ~X, then it’s a fake explanation; their theory really doesn’t tell them what the market will do. If a theory can convincingly explain false data, you’ve got to be very cautious in using it to make predictions or claims about the world.
Conversely, theories with extremely high predictive power will consistently pass the test. If you used facts centered in physics or chemistry, a competent test-taker should always spot the false data, because our theories in physics and chemistry mostly have extremely precise predictive power.
I didn’t understand Psychohistorian’s post as suggesting that we should make up fictional data—for then of course it may be no surprise that the given theory would have to bend in order to accommodate it. Rather, we should take real data, which is not explained by the theory (but which is understood in light of some different theory), and see just how easily the advocate can stretch his explanation to accommodate it. Does he/she notice the stretch? Can he/she resolve the difference between that data from the others?
People get into man-with-a-hammer mode with evolutionary explanations. A lot. Because of the nature of evolutionary biology, sometimes they just reason like, “I can imagine what advantages this feature could have conferred in the past. Thus, …”. And yes, a lot of the time what you get is ad hoc crap.
But what if we don’t know which data is actually explained by the theory or not? That will make it hard to come up with “real data, which is not explained by the theory”.
Not quite. The idea is to see if the theory can convincingly explain fake data. If it can, it doesn’t mean the theory is wrong, it just means your capacity to infer things from it is limited. Natural selection is interesting and useful, but it is not a reliable predictor in many cases. You routinely see people say, “The market must do X because of Y.” If they could say basically the exact same thing about ~X, then it’s a fake explanation; their theory really doesn’t tell them what the market will do. If a theory can convincingly explain false data, you’ve got to be very cautious in using it to make predictions or claims about the world.
Conversely, theories with extremely high predictive power will consistently pass the test. If you used facts centered in physics or chemistry, a competent test-taker should always spot the false data, because our theories in physics and chemistry mostly have extremely precise predictive power.