“So if you wouldn’t sacrifice >0.01AUC for the sake of what a human thinks is the “reasonable” explanation to a problem, in the above thought experiment, then why sacrifice unknown amounts of lost accuracy for the sake of explainability?”
You could think of explainability as some form of regularization to reduce overfitting (to the test set).
For the pedantic among you assume the AUC above is determined via k-fold cross-validation with a very large number of folds and that we don’t mix samples from the same patient between folds
As a general rule of thumb, I agree with you, explainability techniques might often help with generalization. Or at least be intermixed with them. For example, to use techniques that alter the input space, it helps to train with dropout and to have certain activations that could be seen as promoting homogenous behavior on OOD data
“So if you wouldn’t sacrifice >0.01AUC for the sake of what a human thinks is the “reasonable” explanation to a problem, in the above thought experiment, then why sacrifice unknown amounts of lost accuracy for the sake of explainability?”
You could think of explainability as some form of regularization to reduce overfitting (to the test set).
Regrading the thought experiment:
As a general rule of thumb, I agree with you, explainability techniques might often help with generalization. Or at least be intermixed with them. For example, to use techniques that alter the input space, it helps to train with dropout and to have certain activations that could be seen as promoting homogenous behavior on OOD data