I like this overall direction for how simple and robust it is. One challenge I see is that the latent capability of misalignment is still deeply ingrained in the model, and this could be abused by a bad actor even if the model itself doesn’t abuse it. For example, a user could make the model simulate a misaligned human/AI and use the simulated output to drive a local agent chassis. One way around this would be to simply not show misaligned output to the user, but this wouldn’t defend against cases where someone (e.g. a hacker, an employee) gets access to the raw model. A more robust idea I have is to make the simulated humans/AIs fundamentally less capable than the aligned AI. Assuming that the aligned AI is more intelligent than all misaligned humans/AIs (if this wasn’t true, we have bigger problems), the aligned AI only needs to simulate the misaligned human/AI at its (lower) capability level to accurately model the world. This could be done by ensuring that all <AI_quoting_human/AI> examples are strictly at the quoted entity’s capability or by training a less capable “quote model”.
I was indeed assuming that the model is not open-source and that control of the the inference infrastructure is good. So bad actors inside the company, or able to strong-arm the company or penetrate ist security are still a problem, As indeed are bad actors who can develop their own model.
Since my proposed means of control is token based, the required controls could simply be hardwired into the tokenizer code used for inference, so that abuse of the model would require sufficient access to change the tokenizer code. That in turn could be monitored using stabdard software security approaches (access controls, signing, and so forth).
If the model is actually superintelligent, then it generally simulating misaligned humans as being less smart than it is seems likely to be the default behavior, since humans are less smart than it. However, this can probably be bent a little by prompting (John Von Neuman was human, after all), and simulating a large number of misaligned John Von Neuman equivalents all cooperating at high speed (a nation of criminal masterminds in a data center) is not entirely safe, even if we have an even smarter superintelligent aligned AI to try to ride herd/do law enforcement on them.
Simulating anything unaligned as less capable rather remains me of the old crime shows of the 1950s and 1960s where it seemed to be an unwritten rule that the show writers were operating under that all criminals must be stupid and very bad planning skills. Nevertheless, it does seem like a good idea from an alignment point of view, if we could figure out how to implement it. (Fortunately for now all AI has bad planning skills, but that clearly isn’t going to last.)
I like this overall direction for how simple and robust it is. One challenge I see is that the latent capability of misalignment is still deeply ingrained in the model, and this could be abused by a bad actor even if the model itself doesn’t abuse it. For example, a user could make the model simulate a misaligned human/AI and use the simulated output to drive a local agent chassis. One way around this would be to simply not show misaligned output to the user, but this wouldn’t defend against cases where someone (e.g. a hacker, an employee) gets access to the raw model. A more robust idea I have is to make the simulated humans/AIs fundamentally less capable than the aligned AI. Assuming that the aligned AI is more intelligent than all misaligned humans/AIs (if this wasn’t true, we have bigger problems), the aligned AI only needs to simulate the misaligned human/AI at its (lower) capability level to accurately model the world. This could be done by ensuring that all <AI_quoting_human/AI> examples are strictly at the quoted entity’s capability or by training a less capable “quote model”.
I was indeed assuming that the model is not open-source and that control of the the inference infrastructure is good. So bad actors inside the company, or able to strong-arm the company or penetrate ist security are still a problem, As indeed are bad actors who can develop their own model.
Since my proposed means of control is token based, the required controls could simply be hardwired into the tokenizer code used for inference, so that abuse of the model would require sufficient access to change the tokenizer code. That in turn could be monitored using stabdard software security approaches (access controls, signing, and so forth).
If the model is actually superintelligent, then it generally simulating misaligned humans as being less smart than it is seems likely to be the default behavior, since humans are less smart than it. However, this can probably be bent a little by prompting (John Von Neuman was human, after all), and simulating a large number of misaligned John Von Neuman equivalents all cooperating at high speed (a nation of criminal masterminds in a data center) is not entirely safe, even if we have an even smarter superintelligent aligned AI to try to ride herd/do law enforcement on them.
Simulating anything unaligned as less capable rather remains me of the old crime shows of the 1950s and 1960s where it seemed to be an unwritten rule that the show writers were operating under that all criminals must be stupid and very bad planning skills. Nevertheless, it does seem like a good idea from an alignment point of view, if we could figure out how to implement it. (Fortunately for now all AI has bad planning skills, but that clearly isn’t going to last.)