RL has seemed to get a lot greater alignment in sample environments than evolution
The AI-2027 forecasters think otherwise: “Consider this experiment, where a tiny neural net was trained to navigate small virtual mazes to find the ‘cheese’ object. During training, the cheese was always placed somewhere in the top right area of the maze. It seems that the AI did learn a sort of rudimentary goal-directedness–specifically, it learned something like “If not already in the top-right corner region, go towards there; if already there, go towards the cheese.” Part of how we know this is that we can create a test environment where the cheese is somewhere else in the maze, and the AI will ignore the cheese and walk right past it, heading instead towards the top-right corner.”
They also generalize much more intuitively than small RL-from-scratch models, which is probably the more important feature. E.g. an LLM trained in that environment would probably just figure that the cheese was the objective.
The AI-2027 forecasters think otherwise: “Consider this experiment, where a tiny neural net was trained to navigate small virtual mazes to find the ‘cheese’ object. During training, the cheese was always placed somewhere in the top right area of the maze. It seems that the AI did learn a sort of rudimentary goal-directedness–specifically, it learned something like “If not already in the top-right corner region, go towards there; if already there, go towards the cheese.” Part of how we know this is that we can create a test environment where the cheese is somewhere else in the maze, and the AI will ignore the cheese and walk right past it, heading instead towards the top-right corner.”
Way more RL is done on LLMs than tiny neural nets.
They also generalize much more intuitively than small RL-from-scratch models, which is probably the more important feature. E.g. an LLM trained in that environment would probably just figure that the cheese was the objective.