Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier’s.
I don’t think that’s a good assumption. My normal assumption is that if you crank a pretty basic learning algorithm against a learned classifier, you’ll find an adversarial policy in short order. For example, a paper from 2022 achieved “a >97% win rate against KataGo running at superhuman settings … [u]sing less than 14% of the compute used to train KataGo…. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders that cause it to lose the game.”
Also RL & search systems (if they get to AGI) would be able to do “real” open-ended continual learning (see here), whereas (IMO) LLMs are basically stuck with the human conceptual space. That means that an RL & search AGI can rocket way out of distribution, by figuring out new concepts and building new technologies etc.
This causes two problems: (1) the LLM would struggle to judge whether the AGI’s behavior is good or bad, and (2) the LLMs would struggle to judge whether the AGI’s thoughts are good or bad. The latter is important because of the generic issue that (IMO) “Behaviorist” RL reward functions lead to scheming. Judging inscrutable thoughts is hard under the best of circumstances, but when the thoughts might include just-invented new conceptual spaces unlike any that have been thought by anyone before, it’s even harder.
Even in the case of just judging behavior, it’s maybe a bit like taking an LLM trained on pre-1930 text and asking it intricate questions about the ethics of social media meme accounts and so on. Maybe it could do OK if there were an honest broker filling the LLM’s context window with tons of relevant information about what all these different words mean and what the heck is that glowing screen full of rectangles. But there is no honest broker; the AGI itself has a different agenda.
This is good concrete brainstorming, thanks.
I don’t think that’s a good assumption. My normal assumption is that if you crank a pretty basic learning algorithm against a learned classifier, you’ll find an adversarial policy in short order. For example, a paper from 2022 achieved “a >97% win rate against KataGo running at superhuman settings … [u]sing less than 14% of the compute used to train KataGo…. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders that cause it to lose the game.”
Also RL & search systems (if they get to AGI) would be able to do “real” open-ended continual learning (see here), whereas (IMO) LLMs are basically stuck with the human conceptual space. That means that an RL & search AGI can rocket way out of distribution, by figuring out new concepts and building new technologies etc.
This causes two problems: (1) the LLM would struggle to judge whether the AGI’s behavior is good or bad, and (2) the LLMs would struggle to judge whether the AGI’s thoughts are good or bad. The latter is important because of the generic issue that (IMO) “Behaviorist” RL reward functions lead to scheming. Judging inscrutable thoughts is hard under the best of circumstances, but when the thoughts might include just-invented new conceptual spaces unlike any that have been thought by anyone before, it’s even harder.
Even in the case of just judging behavior, it’s maybe a bit like taking an LLM trained on pre-1930 text and asking it intricate questions about the ethics of social media meme accounts and so on. Maybe it could do OK if there were an honest broker filling the LLM’s context window with tons of relevant information about what all these different words mean and what the heck is that glowing screen full of rectangles. But there is no honest broker; the AGI itself has a different agenda.