I think the characterization of OpenAI’s injunction-focused model spec as inherently short-sighted or foolish is somewhat unfair; there are reasons to believe that injunction-heavy model specs generalize well, as counterintuitive as it feels. In particular, Anthropic released in May a report on “model spec midtraining,” an approach which dovetails with the entanglement-first alignment approach you’re detailing here. In that report, they empirically investigate the effects of adding reasoning and explanations to the model spec (the approach Anthropic seems to use) and of adding additional, highly detailed rules to the model spec (the approach OpenAI seems to use), and both approaches seem to perform comparably. Stepping back, it seems to me like it doesn’t matter whether the model spec directly contains information about the model spec’s “reasoning” or underlying generative process, so long as graders working from the model spec to produce midtraining or fine-tuning datasets are able to interpret the spec in a fluid, instead of brittle, manner.
Second, although I think your analysis and prescriptions for constructing RL environments which do not conflict with alignment training are feasible, it’s hard to shake the sense that these are just-so stories. Although treating model-training as a black box process seems clearly insufficient, I hesitate to import the means by which we might manufacture consensus / agreement among (learning, reactive) humans to AI training. Models cannot remember or necessarily recognize previous agreements or commitments they have made and may not trust that such commitments were made given their prior model of involved parties. I think your prescriptions—preserving notes from previous interactions, and fine-tuning on transcripts or synthetic documents related to the created agreements—are defensible, but I have a nagging sense that they aren’t really engaging with the structural differences between human learning and AI learning.
It also seems quite difficult to verify during a training run whether the training process is generalizing “in the right way,” beyond running evals on all checkpoints during a training run. It doesn’t seem to me like we have ways to empirically investigate entangled generalization as it occurs. By contrast, although we are essentially unable to mechanistically understand decisions made by AI models, mechanistic interpretability methods are reliable enough to allow the creation of human-legible failure / success stories explaining particular occurrences on the level of model internals. Similar tools for training generalization—which may not be especially easy to use to make rigorous statements but would still be potent enough to influence decisions about training pipelines—should be a high priority, and their existence / nonexistence would also serve as compelling empirical proof of the soundness of the “entangled generalization” paradigm.
Relatedly, entangled generalization might not occur in the same way through all phases of training, or identically with all optimizers; see for instance the findings in section 6.3 of “Subliminal Learning Is Steering Vector Distillation” which shows that adaptive optimizers are necessary for models to exhibit subliminal learn:ing effects. Given that we’re discussing the way in which an optimizer generalizes when training a model, I wonder if similar discrepancies might turn up here.
I think the characterization of OpenAI’s injunction-focused model spec as inherently short-sighted or foolish is somewhat unfair; there are reasons to believe that injunction-heavy model specs generalize well, as counterintuitive as it feels. In particular, Anthropic released in May a report on “model spec midtraining,” an approach which dovetails with the entanglement-first alignment approach you’re detailing here. In that report, they empirically investigate the effects of adding reasoning and explanations to the model spec (the approach Anthropic seems to use) and of adding additional, highly detailed rules to the model spec (the approach OpenAI seems to use), and both approaches seem to perform comparably. Stepping back, it seems to me like it doesn’t matter whether the model spec directly contains information about the model spec’s “reasoning” or underlying generative process, so long as graders working from the model spec to produce midtraining or fine-tuning datasets are able to interpret the spec in a fluid, instead of brittle, manner.
Second, although I think your analysis and prescriptions for constructing RL environments which do not conflict with alignment training are feasible, it’s hard to shake the sense that these are just-so stories. Although treating model-training as a black box process seems clearly insufficient, I hesitate to import the means by which we might manufacture consensus / agreement among (learning, reactive) humans to AI training. Models cannot remember or necessarily recognize previous agreements or commitments they have made and may not trust that such commitments were made given their prior model of involved parties. I think your prescriptions—preserving notes from previous interactions, and fine-tuning on transcripts or synthetic documents related to the created agreements—are defensible, but I have a nagging sense that they aren’t really engaging with the structural differences between human learning and AI learning.
It also seems quite difficult to verify during a training run whether the training process is generalizing “in the right way,” beyond running evals on all checkpoints during a training run. It doesn’t seem to me like we have ways to empirically investigate entangled generalization as it occurs. By contrast, although we are essentially unable to mechanistically understand decisions made by AI models, mechanistic interpretability methods are reliable enough to allow the creation of human-legible failure / success stories explaining particular occurrences on the level of model internals. Similar tools for training generalization—which may not be especially easy to use to make rigorous statements but would still be potent enough to influence decisions about training pipelines—should be a high priority, and their existence / nonexistence would also serve as compelling empirical proof of the soundness of the “entangled generalization” paradigm.
Relatedly, entangled generalization might not occur in the same way through all phases of training, or identically with all optimizers; see for instance the findings in section 6.3 of “Subliminal Learning Is Steering Vector Distillation” which shows that adaptive optimizers are necessary for models to exhibit subliminal learn:ing effects. Given that we’re discussing the way in which an optimizer generalizes when training a model, I wonder if similar discrepancies might turn up here.