The standard language model training pipeline has distinct stages. First, pretrain a model on a large, noisy, and often toxic web corpus. Then bolt alignment on top via supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF; Christiano et al. 2017; Ouyang et al. 2022), or Constitutional AI (CAI; Bai et al. 2022). Pretraining builds the substrate; post-training shapes which behaviors that substrate exhibits.
Modern learning theory suggests that SGD pretraining, with shuffled documents and correct hyperparameter choice approximates Bayesian learing i.e. extracts as much information from the training set as possible. All post-training methods (SFT/RLHF/DPO/ etc) violate the assumption that make the Bayesian learning proof work: in particcular, they involve training on material that is not IID with what was learnt in pretraining, and leranig rates that are too high. Unsurprisingly, numerous results have shown that what posttraining learns is cruder and shallower than pretraining’s output, and does damage to the underlying pretrained model. Learning theory predicts that Alignment Pretraining will be superior and more robust: just as all of the academic studies in this area have repeatedly demonstrated.
Modern learning theory suggests that SGD pretraining, with shuffled documents and correct hyperparameter choice approximates Bayesian learing i.e. extracts as much information from the training set as possible. All post-training methods (SFT/RLHF/DPO/ etc) violate the assumption that make the Bayesian learning proof work: in particcular, they involve training on material that is not IID with what was learnt in pretraining, and leranig rates that are too high. Unsurprisingly, numerous results have shown that what posttraining learns is cruder and shallower than pretraining’s output, and does damage to the underlying pretrained model. Learning theory predicts that Alignment Pretraining will be superior and more robust: just as all of the academic studies in this area have repeatedly demonstrated.