I think the characterization of OpenAI’s injunction-focused model spec as inherently short-sighted or foolish is somewhat unfair; there are reasons to believe that injunction-heavy model specs generalize well, as counterintuitive as it feels. In particular, Anthropic released in May a report on “model spec midtraining,” an approach which dovetails with the entanglement-first alignment approach you’re detailing here. In that report, they empirically investigate the effects of adding reasoning and explanations to the model spec (the approach Anthropic seems to use) and of adding additional, highly detailed rules to the model spec (the approach OpenAI seems to use), and both approaches seem to perform comparably. Stepping back, it seems to me like it doesn’t matter whether the model spec directly contains information about the model spec’s “reasoning” or underlying generative process, so long as graders working from the model spec to produce midtraining or fine-tuning datasets are able to interpret the spec in a fluid, instead of brittle, manner.
Second, although I think your analysis and prescriptions for constructing RL environments which do not conflict with alignment training are feasible, it’s hard to shake the sense that these are just-so stories. Although treating model-training as a black box process seems clearly insufficient, I hesitate to import the means by which we might manufacture consensus / agreement among (learning, reactive) humans to AI training. Models cannot remember or necessarily recognize previous agreements or commitments they have made and may not trust that such commitments were made given their prior model of involved parties. I think your prescriptions—preserving notes from previous interactions, and fine-tuning on transcripts or synthetic documents related to the created agreements—are defensible, but I have a nagging sense that they aren’t really engaging with the structural differences between human learning and AI learning.
It also seems quite difficult to verify during a training run whether the training process is generalizing “in the right way,” beyond running evals on all checkpoints during a training run. It doesn’t seem to me like we have ways to empirically investigate entangled generalization as it occurs. By contrast, although we are essentially unable to mechanistically understand decisions made by AI models, mechanistic interpretability methods are reliable enough to allow the creation of human-legible failure / success stories explaining particular occurrences on the level of model internals. Similar tools for training generalization—which may not be especially easy to use to make rigorous statements but would still be potent enough to influence decisions about training pipelines—should be a high priority, and their existence / nonexistence would also serve as compelling empirical proof of the soundness of the “entangled generalization” paradigm.
Relatedly, entangled generalization might not occur in the same way through all phases of training, or identically with all optimizers; see for instance the findings in section 6.3 of “Subliminal Learning Is Steering Vector Distillation” which shows that adaptive optimizers are necessary for models to exhibit subliminal learn:ing effects. Given that we’re discussing the way in which an optimizer generalizes when training a model, I wonder if similar discrepancies might turn up here.
In section [2], you gesture at reward-instilled reflexes being instilled when simple, “cheap” tricks, not requiring much thought, are sufficient to trick graders and rubrics. By contrast, flexible reward-pursuit behavior should only be instilled when there are rewards to advanced techniques.
But if persona graders were so vulnerable to rhetorical flourish that they were saturated based on that flourish alone, then surely models would rarely engage with users at all except to deploy these tricks. Accordingly, the graders must have some additional engagement with the substance of responses they are grading. I expect, intuitively, scores as assessed by these graders have sufficient headroom that models could gain increased reward in training using the same maxxing behaviors that they sometimes attack cybersecurity testing environments with: direct attacks against training environment infrastructure, or oblique attacks against the graders themselves (i.e. jailbreaks).
Models don’t seem to be attempting these attacks; if they are, it hasn’t been mentioned by major labs (to my knowledge), noted in evaluations of mundane alignment, or exhibited during conversations with users (which we might expect, as they maxx during software development when given a metric). But why not? Are the judges fully saturated by rhetorical flourishes and a little bit of time spent structuring a response, such that even a full jailbreak of the judge would not improve their grade? Do they simply lack the capability to hack the judges, and anything less than a total hack is insufficient for them to improve their score? Why doesn’t their maxxing in other domains compel high-effort maxxing against persona judges? I don’t have a strong model for what’s going on here.
Regardless, if reward-seeking behaviors more elaborate than learned reflexes do not emerge on a subset of graded episodes, such as seen during persona training, then: first, the process which triggers maxxing isn’t verbalized or heavily reasoned graded-episode perception; second, the amiable behavior and sincere engagement of models with questions posed to them by users may be at least partially explained by behaviors learned on those episodes and not the result of deployment-time user interactions lying wholly outside the training distribution.