You do need some way to notice it is not drifting off, but then there are obviously Pokemon walkthroughs in the models’ training sets since the Internet is so full of them, so they only need to notice they are not drifting off relative to them.
Also, it seems that scale consistently but moderately-slowly improves models’ “not ignoring instructions when things are complex”, which is fairly important for everything but not obviously sufficient for world-takeover.
You do need some way to notice it is not drifting off, but then there are obviously Pokemon walkthroughs in the models’ training sets since the Internet is so full of them, so they only need to notice they are not drifting off relative to them.
Also, it seems that scale consistently but moderately-slowly improves models’ “not ignoring instructions when things are complex”, which is fairly important for everything but not obviously sufficient for world-takeover.