This is cool research. But I want to emphasize that the usage of AUROC for the evaluation of any binary classifier is generally questionable because a high AUROC value only indicates 1) a large true positive rate and 2) a large true negative rate (= low false positive rate).
But a high value of an appropriate statistic for a binary predictor should maximize all four of these values:
A high value of the AUROC only means 1 and 2 are large, while 3 and 4 could be very small. In which case we would clearly have a bad classifier despite a high AUROC value. For details see this paper.
There is another statistic which can be used instead of the AUROC while avoiding this problem: the phi coefficient / “MCC”. This is simply the binary version of the standard Pearson correlation.
A high value of the MCC (close to +1) indicates that all four of the above probabilities are large, and a high negative value (close to −1) that all four are small. If the predictor and the measured variable are independent (the classifier guesses randomly) the value of the MCC is 0.
The linked paper above goes so far as to say
In this short study, we explain why the Matthews correlation coefficient should replace the ROC AUC as standard statistic in all the scientific studies involving a binary classification, in all scientific fields.
This is cool research. But I want to emphasize that the usage of AUROC for the evaluation of any binary classifier is generally questionable because a high AUROC value only indicates 1) a large true positive rate and 2) a large true negative rate (= low false positive rate).
But a high value of an appropriate statistic for a binary predictor should maximize all four of these values:
P(actually true | predicted true) (= true positive rate, recall, sensitivity)
P(actually false | predicted false) (= true negative rate, specificity, converse of false positive rate)
P(predicted true | actually true) (= positive predictive value, precision)
P(predicted false | actually false) (= negative predictive value)
A high value of the AUROC only means 1 and 2 are large, while 3 and 4 could be very small. In which case we would clearly have a bad classifier despite a high AUROC value. For details see this paper.
There is another statistic which can be used instead of the AUROC while avoiding this problem: the phi coefficient / “MCC”. This is simply the binary version of the standard Pearson correlation.
A high value of the MCC (close to +1) indicates that all four of the above probabilities are large, and a high negative value (close to −1) that all four are small. If the predictor and the measured variable are independent (the classifier guesses randomly) the value of the MCC is 0.
The linked paper above goes so far as to say