My point for this post is that we need a plan for RL & search AGI, and that we don’t have one. A plan might involve a certain starting training environment in addition to a certain reward function. Sure.
If you’re suggesting that, as long as the training environment is good (e.g. a loving family with adequate socializing), then we can use some straightforward reward function (e.g. “reward when the supervisor presses the ‘approve’ button”), and the AI will still wind up being genuinely nice, then I strongly disagree. E.g. plenty of human sociopaths grow up in loving families; a paperclip maximizer would maximize paperclips regardless of its childhood environment; and when people raise undomesticated animals as beloved pets, they often wind up getting mauled.
So I think the reward function has to be a central part of the plan, and I tend to emphasize that. But yeah, whenever I say “What reward function would lead to a nice AGI?”, that’s shorthand for (e.g. here) “What reward function (along with training environment and other design choices) would lead to a nice AGI?”.
people, unlike LLMs, don’t tend to imitate their enemies. I wonder how it interacts with tropes like The Chain of Harm and bullying traditions
This is kinda off-topic, but I’ll respond anyway.
I think the motivation to act aggressively (in certain contexts) is innate in humans, just as it’s innate in probably all complex animals. So we don’t need to explain from scratch how people develop a motivation to bully each other. The motivation is already there, and might or might not come out depending on their personality, their mood, their relationship to the potential target, and lots of other mitigating and aggravating factors.
By contrast, we do need to explain how someone might develop a motivation to, I dunno, get a cupcake tattoo. There’s no innate drive that makes it feel satisfying to get a cupcake tattoo. Instead, the explanation would probably be that a cupcake tattoo is the kind of thing that this person’s idols (real or imagined) would think is cool.
My main issue is with your post as-written is the response to question Q6. The alternative C has researchers attempt to maximize the output of a learned classifier, causing the optimum to go out of distribution into pathological craziness. Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier’s. For example, if the ECI of the classifying LLM is , while the smallest ECI causing pathological madness is then an AI with ECI of becomes able to trust the AI who was trained to optimize its judgement until reaching the ECI of , who in turn can trust the AI with the ECI of , etc.
Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier’s.
I don’t think that’s a good assumption. My normal assumption is that if you crank a pretty basic learning algorithm against a learned classifier, you’ll find an adversarial policy in short order. For example, a paper from 2022 achieved “a >97% win rate against KataGo running at superhuman settings … [u]sing less than 14% of the compute used to train KataGo…. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders that cause it to lose the game.”
Also RL & search systems (if they get to AGI) would be able to do “real” open-ended continual learning (see here), whereas (IMO) LLMs are basically stuck with the human conceptual space. That means that an RL & search AGI can rocket way out of distribution, by figuring out new concepts and building new technologies etc.
This causes two problems: (1) the LLM would struggle to judge whether the AGI’s behavior is good or bad, and (2) the LLMs would struggle to judge whether the AGI’s thoughts are good or bad. The latter is important because of the generic issue that (IMO) “Behaviorist” RL reward functions lead to scheming. Judging inscrutable thoughts is hard under the best of circumstances, but when the thoughts might include just-invented new conceptual spaces unlike any that have been thought by anyone before, it’s even harder.
Even in the case of just judging behavior, it’s maybe a bit like taking an LLM trained on pre-1930 text and asking it intricate questions about the ethics of social media meme accounts and so on. Maybe it could do OK if there were an honest broker filling the LLM’s context window with tons of relevant information about what all these different words mean and what the heck is that glowing screen full of rectangles. But there is no honest broker; the AGI itself has a different agenda.
My point for this post is that we need a plan for RL & search AGI, and that we don’t have one. A plan might involve a certain starting training environment in addition to a certain reward function. Sure.
If you’re suggesting that, as long as the training environment is good (e.g. a loving family with adequate socializing), then we can use some straightforward reward function (e.g. “reward when the supervisor presses the ‘approve’ button”), and the AI will still wind up being genuinely nice, then I strongly disagree. E.g. plenty of human sociopaths grow up in loving families; a paperclip maximizer would maximize paperclips regardless of its childhood environment; and when people raise undomesticated animals as beloved pets, they often wind up getting mauled.
So I think the reward function has to be a central part of the plan, and I tend to emphasize that. But yeah, whenever I say “What reward function would lead to a nice AGI?”, that’s shorthand for (e.g. here) “What reward function (along with training environment and other design choices) would lead to a nice AGI?”.
See Intro series §12.5 for further discussion.
This is kinda off-topic, but I’ll respond anyway.
I think the motivation to act aggressively (in certain contexts) is innate in humans, just as it’s innate in probably all complex animals. So we don’t need to explain from scratch how people develop a motivation to bully each other. The motivation is already there, and might or might not come out depending on their personality, their mood, their relationship to the potential target, and lots of other mitigating and aggravating factors.
By contrast, we do need to explain how someone might develop a motivation to, I dunno, get a cupcake tattoo. There’s no innate drive that makes it feel satisfying to get a cupcake tattoo. Instead, the explanation would probably be that a cupcake tattoo is the kind of thing that this person’s idols (real or imagined) would think is cool.
My main issue is with your post as-written is the response to question Q6. The alternative C has researchers attempt to maximize the output of a learned classifier, causing the optimum to go out of distribution into pathological craziness. Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier’s. For example, if the ECI of the classifying LLM is
This is good concrete brainstorming, thanks.
I don’t think that’s a good assumption. My normal assumption is that if you crank a pretty basic learning algorithm against a learned classifier, you’ll find an adversarial policy in short order. For example, a paper from 2022 achieved “a >97% win rate against KataGo running at superhuman settings … [u]sing less than 14% of the compute used to train KataGo…. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders that cause it to lose the game.”
Also RL & search systems (if they get to AGI) would be able to do “real” open-ended continual learning (see here), whereas (IMO) LLMs are basically stuck with the human conceptual space. That means that an RL & search AGI can rocket way out of distribution, by figuring out new concepts and building new technologies etc.
This causes two problems: (1) the LLM would struggle to judge whether the AGI’s behavior is good or bad, and (2) the LLMs would struggle to judge whether the AGI’s thoughts are good or bad. The latter is important because of the generic issue that (IMO) “Behaviorist” RL reward functions lead to scheming. Judging inscrutable thoughts is hard under the best of circumstances, but when the thoughts might include just-invented new conceptual spaces unlike any that have been thought by anyone before, it’s even harder.
Even in the case of just judging behavior, it’s maybe a bit like taking an LLM trained on pre-1930 text and asking it intricate questions about the ethics of social media meme accounts and so on. Maybe it could do OK if there were an honest broker filling the LLM’s context window with tons of relevant information about what all these different words mean and what the heck is that glowing screen full of rectangles. But there is no honest broker; the AGI itself has a different agenda.