ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.[9]
This is not a particularly crucial point, but I would categorise this as unambiguously under the umbrella “understanding and shaping ML generalization” and if for some reason I’d stopped reading before this section I would have been quite confident you were not doing anything like this.
I wouldn’t be that surprised if your agenda ends up converging, or in productive dialogue, with other “understand generalization” agendas.
Your reading makes sense and I don’t want to quibble over semantics (or foreclose potentially productive dialog). I do think there’s an important distinction between:
Optimize a loss function that captures performance on the training distribution, and then shape the architecture / curriculum / regularizer / etc. so that the model behaves in a desirable way out of distribution.
Define a loss function that captures performance off of the training distribution (e.g. what would happen in conditionals that we can’t sample in the lab) and optimize that.
I was describing ARC as trying to pursue option #2.
That said, in reality we are going to need to characterize the real world using learned hypotheses, since we can’t give a formal characterization without using ML. And then of course we care about how those hypotheses generalize. So regardless of the distinction above I do think it’s fair to characterize our work as an example of “understanding and shaping ML generalization.” We’re just willing to make much larger changes to get the generalization we want. In particular we are not trying to understand or shape how an agent generalizes, but instead are trying to separate the learning of hypotheses about the data (which do need to generalize well) from mechanistic analyses of predictors and agents that behave well under those hypotheses.
What I really care about is that our methods would likely work even if competitive RL converges robustly to training-gaming behavior, while I’m claiming that other approaches rest on the empirical bet that what generalization you get is sensitive to some training parameters that we can tweak without big performance hits.
This is not a particularly crucial point, but I would categorise this as unambiguously under the umbrella “understanding and shaping ML generalization” and if for some reason I’d stopped reading before this section I would have been quite confident you were not doing anything like this.
I wouldn’t be that surprised if your agenda ends up converging, or in productive dialogue, with other “understand generalization” agendas.
It does seem like a promising line of research.
Your reading makes sense and I don’t want to quibble over semantics (or foreclose potentially productive dialog). I do think there’s an important distinction between:
Optimize a loss function that captures performance on the training distribution, and then shape the architecture / curriculum / regularizer / etc. so that the model behaves in a desirable way out of distribution.
Define a loss function that captures performance off of the training distribution (e.g. what would happen in conditionals that we can’t sample in the lab) and optimize that.
I was describing ARC as trying to pursue option #2.
That said, in reality we are going to need to characterize the real world using learned hypotheses, since we can’t give a formal characterization without using ML. And then of course we care about how those hypotheses generalize. So regardless of the distinction above I do think it’s fair to characterize our work as an example of “understanding and shaping ML generalization.” We’re just willing to make much larger changes to get the generalization we want. In particular we are not trying to understand or shape how an agent generalizes, but instead are trying to separate the learning of hypotheses about the data (which do need to generalize well) from mechanistic analyses of predictors and agents that behave well under those hypotheses.
What I really care about is that our methods would likely work even if competitive RL converges robustly to training-gaming behavior, while I’m claiming that other approaches rest on the empirical bet that what generalization you get is sensitive to some training parameters that we can tweak without big performance hits.