Our general intelligence does not appear to work like that. The AGI we build could and even may look like that; I’m not sure.
Our brains don’t really pass subgoals to submodules or subagents. Instead we turn most of our mind (global workspace, closely tied higher-level cortical regions) to the subgoal while we’re working on it. It’s a serial approach. This seems pretty clear from neural recording data in humans and finer-grained data from animals. And from indirect reasoning from behavioral data. Humans can parallelize tasks reasonably efficiently only when they’re in totally different sensorimotor domains, so separate brain regions can work on them. And even doing this efficiently takes a ton of practice. It’s not default brain function.
So the same reward signals are training your “general intelligence” as training your “subgoal solver”. Sometimes people do sort of get hacked by subgoals, when they become obsessed with a challenge. They usually recover when over practical matters (food, shelter, social approval) become pressing enough that those reward signals dominate over the predicted reward from solving those problems.
LLMs currently work like this too. An LLM AGI might spin out subagents for subgoals, and thereby face the problem you mention. Or it might not; it would be more straightforward to mostly stay single-tracked and rely on subagents only for particularly parallelizable subgoals.
If ASI does use a parallel approach like you mention, we or our AIs will have to solve the alignment risks of the central general intelligence being manipulated/hacked by subagents.
John’s point applies even if you are doing subgoal solving serially. When trying to get food, I don’t naturally consider plans like “shove my fingers down my throat to induce vomiting, so that I’m hungrier and thus buy more food”—even bulimics aren’t inducing vomiting as a plot to get their future self to eat more, I don’t think!
However, I will consider plans like “tie a string around my finger so I remember to buy food” or “set a reminder”.
The difference of course is that I don’t like vomiting and think it’s unhealthy. When I turn my cognition towards solving the problem of “buy food”, this time slice of me still remains ‘aligned’. This is true even when I do complicated metareasoning about my memory, attention, and “external cognition” (by my phone) - I have no problem planning around my memory or attention, and have thoughts like “I shouldn’t open chats right now, because then I’ll get sucked in and not buy food”.
This isn’t about how good I am at planning around myself; it’s about the fact I seriously consider such plans at all, but don’t seriously consider vomiting. As you say, though you sometimes have different values in the moment, we still need to explain how you can ever be ‘corrigible’ to the rest of your problem solving process (I’m not tied to the frame of corrigibility).
Our general intelligence does not appear to work like that. The AGI we build could and even may look like that; I’m not sure.
Our brains don’t really pass subgoals to submodules or subagents. Instead we turn most of our mind (global workspace, closely tied higher-level cortical regions) to the subgoal while we’re working on it. It’s a serial approach. This seems pretty clear from neural recording data in humans and finer-grained data from animals. And from indirect reasoning from behavioral data. Humans can parallelize tasks reasonably efficiently only when they’re in totally different sensorimotor domains, so separate brain regions can work on them. And even doing this efficiently takes a ton of practice. It’s not default brain function.
So the same reward signals are training your “general intelligence” as training your “subgoal solver”. Sometimes people do sort of get hacked by subgoals, when they become obsessed with a challenge. They usually recover when over practical matters (food, shelter, social approval) become pressing enough that those reward signals dominate over the predicted reward from solving those problems.
LLMs currently work like this too. An LLM AGI might spin out subagents for subgoals, and thereby face the problem you mention. Or it might not; it would be more straightforward to mostly stay single-tracked and rely on subagents only for particularly parallelizable subgoals.
If ASI does use a parallel approach like you mention, we or our AIs will have to solve the alignment risks of the central general intelligence being manipulated/hacked by subagents.
John’s point applies even if you are doing subgoal solving serially. When trying to get food, I don’t naturally consider plans like “shove my fingers down my throat to induce vomiting, so that I’m hungrier and thus buy more food”—even bulimics aren’t inducing vomiting as a plot to get their future self to eat more, I don’t think!
However, I will consider plans like “tie a string around my finger so I remember to buy food” or “set a reminder”.
The difference of course is that I don’t like vomiting and think it’s unhealthy. When I turn my cognition towards solving the problem of “buy food”, this time slice of me still remains ‘aligned’. This is true even when I do complicated metareasoning about my memory, attention, and “external cognition” (by my phone) - I have no problem planning around my memory or attention, and have thoughts like “I shouldn’t open chats right now, because then I’ll get sucked in and not buy food”.
This isn’t about how good I am at planning around myself; it’s about the fact I seriously consider such plans at all, but don’t seriously consider vomiting. As you say, though you sometimes have different values in the moment, we still need to explain how you can ever be ‘corrigible’ to the rest of your problem solving process (I’m not tied to the frame of corrigibility).