> The fact that you choose an algorithm does not effect its performance, and you don’t have to worry about Causal Goodhart.
But now, I think you have to worry about a “Regressional Goodhart”
Maybe this would be pedantic to point out, but your choice of the best-performing model on test data is likely to have done that well by chance, as the number of models evaluated increases (hence validation and test splits).
> The fact that you choose an algorithm does not effect its performance, and you don’t have to worry about Causal Goodhart.
But now, I think you have to worry about a “Regressional Goodhart”
Maybe this would be pedantic to point out, but your choice of the best-performing model on test data is likely to have done that well by chance, as the number of models evaluated increases (hence validation and test splits).