We had a conversation in private messages to clarify some of this. I thought I’d publicly post one of my responses:
AI-2027 strategy
Step 1: Train and deploy Safer-1, a misaligned but controlled autonomous researcher. It’s controlled because it’s transparent to human overseers: it uses English chains of thought (CoT) to think, and faithful CoT techniques have been employed to eliminate euphemisms, steganography, and subtle biases.
Step 2: Try out different training environments for Safer-1, and carefully read the CoT to determine the ways in which the goals and principles in the Spec did or didn’t “stick.”
Step 3: Train and deploy Safer-2, an aligned and controlled autonomous researcher based on the same architecture but with a better training environment that incentivizes the right goals and principles this time.
<...>
Step 4: Design, train, and deploy Safer-3, a much smarter autonomous researcher which uses a more advanced architecture similar to the old Agent-4. It’s no longer transparent to human overseers, but it’s transparent to Safer-2. So it should be possible to figure out how to make it both aligned and controlled.
Step 5: Repeat Step 4 ad infinitum, creating a chain of ever-more-powerful, ever-more-aligned AIs that are overseen by the previous links in the chain (e.g. the analogues of Agent-5 from the other scenario branch).
My post is mainly about how this step breaks:
Try out different training environments for Safer-1, and carefully read the CoT to determine the ways in which the goals and principles in the Spec did or didn’t “stick.”
This process leads to finding a set of training environments that lead an AI to have a corrible-looking CoT. It doesn’t lead to finding training environments that produce an AI that is truly corrigible.
So Safer-2 has been shaped by a process that makes one good at looking corrigible. It probably isn’t corrigible, because there’s lots of ways to be non-corrigible but corrigible-looking-to-the-design-process.
Step 4 involves putting a huge amount of trust in Safer-2. Since Safer-2 is untrustworthy, it betrays us in a way that isn’t visible to us (which is extremely straightforward in this scenario because we’ve assumed there are important things that Safer-2 is managing that humans don’t understand).
We had a conversation in private messages to clarify some of this. I thought I’d publicly post one of my responses:
AI-2027 strategy
Step 1: Train and deploy Safer-1, a misaligned but controlled autonomous researcher. It’s controlled because it’s transparent to human overseers: it uses English chains of thought (CoT) to think, and faithful CoT techniques have been employed to eliminate euphemisms, steganography, and subtle biases.
Step 2: Try out different training environments for Safer-1, and carefully read the CoT to determine the ways in which the goals and principles in the Spec did or didn’t “stick.”
Step 3: Train and deploy Safer-2, an aligned and controlled autonomous researcher based on the same architecture but with a better training environment that incentivizes the right goals and principles this time.
<...>
Step 4: Design, train, and deploy Safer-3, a much smarter autonomous researcher which uses a more advanced architecture similar to the old Agent-4. It’s no longer transparent to human overseers, but it’s transparent to Safer-2. So it should be possible to figure out how to make it both aligned and controlled.
Step 5: Repeat Step 4 ad infinitum, creating a chain of ever-more-powerful, ever-more-aligned AIs that are overseen by the previous links in the chain (e.g. the analogues of Agent-5 from the other scenario branch).
My post is mainly about how this step breaks:
This process leads to finding a set of training environments that lead an AI to have a corrible-looking CoT. It doesn’t lead to finding training environments that produce an AI that is truly corrigible.
So Safer-2 has been shaped by a process that makes one good at looking corrigible. It probably isn’t corrigible, because there’s lots of ways to be non-corrigible but corrigible-looking-to-the-design-process.
Step 4 involves putting a huge amount of trust in Safer-2. Since Safer-2 is untrustworthy, it betrays us in a way that isn’t visible to us (which is extremely straightforward in this scenario because we’ve assumed there are important things that Safer-2 is managing that humans don’t understand).