The only thing that’s really different about pretraining and posttraining is the extent of actions available to manipulate the world in advance of it being graded. If you pretrain a model hard enough, it seems conceptually plausible to me that you might get a little AGI inside figuring out the next tokens, which could be dangerous in the same way that running an AI inside a “secure sandbox” is dangerous.
The extent of the actions available to manipulate the world is one of the things that’s different about pretraining and posttraining, but I think other major differences are
Pretraining has much, much higher feedback density than posttraining. With sft on text you’ve got one item of feedback (“was correct y/n”) at every single token position. If you’re distilling from a teacher model you might have even more feedback density than that. Posttraining you get a few items of feedback per trajectory, or maybe a few hundred if you’re lucky and the problem you’re working on affords that.
The next token prediction just has so much headroom for learning simple patterns that scheming/hacking type actions are unlikely to be worth their parameter cost on most natural language datasets
that said if you RL’d an existing reasoning model with tools hard enough on the next token prediction task on a dataset where those tools allowed it to discover or manipulate the next ground truth token through a discoverable path I expect you would see undesired behaviors emerge.
The only thing that’s really different about pretraining and posttraining is the extent of actions available to manipulate the world in advance of it being graded. If you pretrain a model hard enough, it seems conceptually plausible to me that you might get a little AGI inside figuring out the next tokens, which could be dangerous in the same way that running an AI inside a “secure sandbox” is dangerous.
The extent of the actions available to manipulate the world is one of the things that’s different about pretraining and posttraining, but I think other major differences are
Pretraining has much, much higher feedback density than posttraining. With sft on text you’ve got one item of feedback (“was correct y/n”) at every single token position. If you’re distilling from a teacher model you might have even more feedback density than that. Posttraining you get a few items of feedback per trajectory, or maybe a few hundred if you’re lucky and the problem you’re working on affords that.
The next token prediction just has so much headroom for learning simple patterns that scheming/hacking type actions are unlikely to be worth their parameter cost on most natural language datasets
that said if you RL’d an existing reasoning model with tools hard enough on the next token prediction task on a dataset where those tools allowed it to discover or manipulate the next ground truth token through a discoverable path I expect you would see undesired behaviors emerge.