Overfitting is a problem of “thinking” the data given is more strongly determined than it is. Hindsight bias seems similar—we feel that things couldn’t have turned out other than the way they actually did.
Just as “Does induction work?” and “Why does induction work?” are two different questions, we can distinguish the questions “Do people fail to seek alternative explanations?” and “Why do people fail to seek alternative explanations?” The answer to the first is quite obviously “Yes,” and the second is harder to answer, as questions of that form often are.
Before trying to answer it, it seems like a good idea to point out that overfitting is a simple name for a complex phenomenon, and that overfitting as a mistake is probably overdetermined. Statistical inference in general seems far more cognitively complex than the tasks issued to subjects in hindsight bias experiments. So there may very well be multiple explanations to the question “Why do people overfit the data?”
But, I agree with you that both phenomena seem like an example of a failure to seek alternative explanations; specifically, a failure based on the quiet inference that seeking an alternative explanation doesn’t seem necessary in each case.
We see in the article that people infer from the difficulty of seeking alternative explanations that those alternatives are less plausible and that their focal explanation is more plausible. We also see that when you make them discount the relevance of this difficulty, thinking of alternatives has the effect that we initially and naively thought that it would: the more alternatives you imagine, the less determined the past seems.
It seems the reason overfitting happens at all is because there is no clear reason at the time to seek an alternative explanation, besides the outside view. “But it fits so well!” the one says. The experience is so very fluent. What is there to discourage the statistician? Nothing until they use the model on untrained data. They believe that the model is accurate right up until the moment that their perception of the model becomes disfluent.
And at this point it begins to look related to the hindsight bias experiments, at least to me. But I also don’t think that they are especially related, because my answer to a question like “Is overfitting related to availability bias?” or “Is overfitting related to the planning fallacy?” would probably be quite similar. I would maintain that it’s the deep cog-sci results about the effect of phenomenal experiences on judgment that are the important relation, and not the more superficial details like whether or not the task has to do with inventing alternative hypotheses.
Hm. I tend to think about overfitting as a mistake in picking the level of complexity and about the hindsight bias as a mistake about (prior) probabilities. However you have a point: if you have a choice of models to forecast the future and you pick a particular model while suffering from the hindsight bias, this can be seen as overfitting: you “in-sample” error will be low, but your out-of-sample error will be high.
Another way to look at this is to treat it as confusion between a sample estimate and a true population value. The hindsight bias will tell you that the particular realization that you’re observing (=sample) is how it should have been and always will be (=population).
Why do you wonder that? If you care to elaborate.
Overfitting is a problem of “thinking” the data given is more strongly determined than it is. Hindsight bias seems similar—we feel that things couldn’t have turned out other than the way they actually did.
Just as “Does induction work?” and “Why does induction work?” are two different questions, we can distinguish the questions “Do people fail to seek alternative explanations?” and “Why do people fail to seek alternative explanations?” The answer to the first is quite obviously “Yes,” and the second is harder to answer, as questions of that form often are.
Before trying to answer it, it seems like a good idea to point out that overfitting is a simple name for a complex phenomenon, and that overfitting as a mistake is probably overdetermined. Statistical inference in general seems far more cognitively complex than the tasks issued to subjects in hindsight bias experiments. So there may very well be multiple explanations to the question “Why do people overfit the data?”
But, I agree with you that both phenomena seem like an example of a failure to seek alternative explanations; specifically, a failure based on the quiet inference that seeking an alternative explanation doesn’t seem necessary in each case.
We see in the article that people infer from the difficulty of seeking alternative explanations that those alternatives are less plausible and that their focal explanation is more plausible. We also see that when you make them discount the relevance of this difficulty, thinking of alternatives has the effect that we initially and naively thought that it would: the more alternatives you imagine, the less determined the past seems.
I haven’t gotten into it yet, but we use these phenomenal experiences of ease and difficulty to make many, many other judgments: judgments of truth, credibility, beauty, frequency, familiarity, etc. A particularly interesting result is that merely writing a misleading question in a difficult-to-read font is enough to increase the probability that the subject will answer the question correctly.
It seems the reason overfitting happens at all is because there is no clear reason at the time to seek an alternative explanation, besides the outside view. “But it fits so well!” the one says. The experience is so very fluent. What is there to discourage the statistician? Nothing until they use the model on untrained data. They believe that the model is accurate right up until the moment that their perception of the model becomes disfluent.
And at this point it begins to look related to the hindsight bias experiments, at least to me. But I also don’t think that they are especially related, because my answer to a question like “Is overfitting related to availability bias?” or “Is overfitting related to the planning fallacy?” would probably be quite similar. I would maintain that it’s the deep cog-sci results about the effect of phenomenal experiences on judgment that are the important relation, and not the more superficial details like whether or not the task has to do with inventing alternative hypotheses.
Hopefully that makes sense.
Hm. I tend to think about overfitting as a mistake in picking the level of complexity and about the hindsight bias as a mistake about (prior) probabilities. However you have a point: if you have a choice of models to forecast the future and you pick a particular model while suffering from the hindsight bias, this can be seen as overfitting: you “in-sample” error will be low, but your out-of-sample error will be high.
Another way to look at this is to treat it as confusion between a sample estimate and a true population value. The hindsight bias will tell you that the particular realization that you’re observing (=sample) is how it should have been and always will be (=population).