The problem is that you invoke the idea that it’s starting from something close to pareto-optimal. But pareto optimal with respect to what? Pareto optimality implies a multi-objective problem, and it’s not clear what those objectives are. That’s why we need the whole causality framework: the multiple objectives are internal nodes of the DAG.
The standard description of overfitting does fit into the DAG model, but most of the usual solutions to that problem are specific to overfitting; they don’t generalize to Goodhart problems in e.g. management.
I assumed (close to) pareto-optimality, since the OP suggests that most real systems are starting from this state.
The (immediately disceranable) competing objectives here are training error and test error. Only one can be observed ahead of deployment (much like X in the X+Y example earlier), while it’s actually the other which matters. That is not to say that there aren’t other currently undiscovered / unstated metrics of interest (training time, designer time, model size, etc.) which may be articulated and optimised for, still leading to a suboptimal result on test error. Indeed, we can imagine a situation with a perfectly good predictive neural network, which for some reason won’t run on the new hardware that’s provisioned, and so a hasty, over-extended engineer might simply delete entire blocks of it, optimising their time and the ability of the model to fit on a Raspberry PI, while most likely completely voiding any ability of the network to perform the task meaningfully.
If this sounds contrived, forgive me. Perhaps I’m am talking tangentially past the discussion at hand; if so, kindly ignore. Mostly, I only wish to propose that a fundamental formulation of ML, of minimising training loss while we want to reduce test loss, is an example of Goodhart’s law in action, and there is rich literature on techniques to circumvent its effects. Do you agree? Why / why not?
The problem is that you invoke the idea that it’s starting from something close to pareto-optimal. But pareto optimal with respect to what? Pareto optimality implies a multi-objective problem, and it’s not clear what those objectives are. That’s why we need the whole causality framework: the multiple objectives are internal nodes of the DAG.
The standard description of overfitting does fit into the DAG model, but most of the usual solutions to that problem are specific to overfitting; they don’t generalize to Goodhart problems in e.g. management.
I assumed (close to) pareto-optimality, since the OP suggests that most real systems are starting from this state.
The (immediately disceranable) competing objectives here are training error and test error. Only one can be observed ahead of deployment (much like X in the X+Y example earlier), while it’s actually the other which matters. That is not to say that there aren’t other currently undiscovered / unstated metrics of interest (training time, designer time, model size, etc.) which may be articulated and optimised for, still leading to a suboptimal result on test error. Indeed, we can imagine a situation with a perfectly good predictive neural network, which for some reason won’t run on the new hardware that’s provisioned, and so a hasty, over-extended engineer might simply delete entire blocks of it, optimising their time and the ability of the model to fit on a Raspberry PI, while most likely completely voiding any ability of the network to perform the task meaningfully.
If this sounds contrived, forgive me. Perhaps I’m am talking tangentially past the discussion at hand; if so, kindly ignore. Mostly, I only wish to propose that a fundamental formulation of ML, of minimising training loss while we want to reduce test loss, is an example of Goodhart’s law in action, and there is rich literature on techniques to circumvent its effects. Do you agree? Why / why not?