Regarding the cost of a making an incorrect probability estimate, “Overconfidence is just as bad as underconfidence.” is not generally true. In binary classification contexts, one leads to more false positives and another to more false negatives. The costs of each are not equal in general for real world situations.
The author may simply mean that both are incorrect; this I accept.
My point is more than pedantic; there are too many examples of machine learning systems failing to recognize different misclassification costs.
Regarding the cost of a making an incorrect probability estimate, “Overconfidence is just as bad as underconfidence.” is not generally true. In binary classification contexts, one leads to more false positives and another to more false negatives. The costs of each are not equal in general for real world situations.
The author may simply mean that both are incorrect; this I accept.
My point is more than pedantic; there are too many examples of machine learning systems failing to recognize different misclassification costs.