Why did you write “This post [Inaccessible Information] doesn’t reflect me becoming more pessimistic about iterated amplification or alignment overall.” just one month before publishing “Learning the prior”? (Is it because you were classifying “learning the prior” / imitative generalization under “iterated amplification” and now you consider it a different algorithm?)
For example, at the beginning of modern cryptography you could describe the methodology as “Tell a story about how someone learns something about your secret” and that only gradually crystallized into definitions like semantic security (and still people sometimes retreat to this informal process in order to define and clarify new security notions).
Why doesn’t the analogy with cryptography make you a lot more pessimistic about AI alignment, as it did for me?
The best case is that we end up with a precise algorithm for which we still can’t tell any failure story. In that case we should implement it (in some sense this is just the final step of making it precise) and see how it works in practice.
Would you do anything else to make sure it’s safe, before letting it become potentially superintelligent? For example would you want to see “alignment proofs” similar to “security proofs” in cryptography? What if such things do not seem feasible or you can’t reach very high confidence that the definitions/assumptions/proofs are correct?
I would also expect that e.g. if you were to describe almost any existing practical system with purported provable security, it would be straightforward for a layperson with theoretical background (e.g. me) to describe possible attacks that are not precluded by the security proof, and that it wouldn’t even take that long.
I guess SSH itself would be an interesting test of this, e.g. comparing the theoretical model of this paper to a modern implementation. What is your view about that comparison? e.g. how do you think about the following possibilities:
There is no material weakness in the security proof.
A material weakness is already known.
An interested layperson could find a material weakness with moderate effort.
An expert could find a material weakness with significant effort.
My guess would be that probably we’re in world 2, and if not that it’s probably because no one cares that much (e.g. because it’s obvious that there will be some material weakness and the standards of the field are such that it’s not publishable unless it actually comes with an attack) and we are in world 3.
(On a quick skim, and from the author’s language when describing the model, my guess is that material weaknesses of the model are more or less obvious and that the authors are aware of potential attacks not covered by their model.)
I’m still curious for your view on the crypto examples you cited. My current understanding is that people do not expect the security proofs to rule out all possible attacks (a situation I can sympathize with since I’ve written multiple proofs that rule out large classes of attacks without attempting to cover all possible attacks), so I’m interested in whether (i) you disagree with that and believe that serious onlookers have had the expectation that proofs are comprehensive, (ii) you agree but feel it would be impractical to give a correct proof and this is a testament to the difficulty of proving things, (iii) you feel it would be possible but prohibitively expensive, and are expressing a quantitative point about the cost of alignment analyses being impractical, (iv) you feel that the crypto case would be practical but the AI case is likely to be much harder and just want to make a directionally analogous update.
I still feel like more of the action is in my skepticism about the (alignment analysis) <--> (security analysis) analogy, but I could still get some update out of the analogy if the crypto situation is thornier than I currently believe.
Why did you write “This post [Inaccessible Information] doesn’t reflect me becoming more pessimistic about iterated amplification or alignment overall.” just one month before publishing “Learning the prior”? (Is it because you were classifying “learning the prior” / imitative generalization under “iterated amplification” and now you consider it a different algorithm?)
I think that post is basically talking about the same kinds of hard cases as in Towards Formalizing Universality 1.5 years earlier (in section IV), so it’s intended to be more about clarification/exposition than changing views.
See the thread with Rohin above for some rough history.
Why doesn’t the analogy with cryptography make you a lot more pessimistic about AI alignment, as it did for me?
I’m not sure.It’s possible I would become more pessimistic if I walked through concrete cases of people’s analyses being wrong in subtle and surprising ways.
My experience with practical systems is that it is usually easy for theorists to describe hypothetical breaks for the security model, and the issue is mostly one of prioritization (since people normally don’t care too much about security). For example, my strong expectation would be that people had described hypothetical attacks on any of the systems discussed in the article you linked prior to their implementation, at least if they had ever been subject to formal scrutiny. The failures are just quite far away from the levels of paranoia that I’ve seen people on the theory side exhibit when they are trying to think of attacks.
I would also expect that e.g. if you were to describe almost any existing practical system with purported provable security, it would be straightforward for a layperson with theoretical background (e.g. me) to describe possible attacks that are not precluded by the security proof, and that it wouldn’t even take that long. It sounds like a fun game.
Another possible divergence is that I’m less convinced by the analogy, since alignment seems more about avoiding the introduction of adversarial consequentialists and it’s not clear if that game behaves in the same way. I’m not sure if that’s more or less important than the prior point.
Would you do anything else to make sure it’s safe, before letting it become potentially superintelligent? For example would you want to see “alignment proofs” similar to “security proofs” in cryptography?
I would want to do a lot of work before deploying an algorithm in any context where a failure would be catastrophic (though “before letting it become potentially superintelligent” kind of suggests a development model I’m not on board with).
That would ideally involve theoretical analysis from a lot of angles, e.g. proofs of key properties that are amenable to proof, demonstrations of how the system could plausibly fail if we were wrong about key claims or if we relax assumptions, and so on.
It would also involve good empirical characterization, including things like running on red team inputs, or changing the training procedure in ways that seem as bad as possible while still preserving our alignment arguments, and performing extensive evals under those more pessimistic conditions. It would involve validating key claims individually, and empirically testing other claims that are established by structurally similar arguments. It would involve characterizing scaling behavior where applicable and understanding it as well as we can (along with typical levels of variability and plausible stories about deviations from trend).
What if such things do not seem feasible or you can’t reach very high confidence that the definitions/assumptions/proofs are correct?
I’m not exactly sure what you are asking. It seems like we’ll do what we can on all the fronts and prioritize them as well as we can. Do you mean, what else can we say today about what methodologies we’d use? Or under what conditions would I pivot to spending down my political capital to delay deployment? Or something else?
Why did you write “This post [Inaccessible Information] doesn’t reflect me becoming more pessimistic about iterated amplification or alignment overall.” just one month before publishing “Learning the prior”? (Is it because you were classifying “learning the prior” / imitative generalization under “iterated amplification” and now you consider it a different algorithm?)
Why doesn’t the analogy with cryptography make you a lot more pessimistic about AI alignment, as it did for me?
Would you do anything else to make sure it’s safe, before letting it become potentially superintelligent? For example would you want to see “alignment proofs” similar to “security proofs” in cryptography? What if such things do not seem feasible or you can’t reach very high confidence that the definitions/assumptions/proofs are correct?
In my other response to your comment I wrote:
I guess SSH itself would be an interesting test of this, e.g. comparing the theoretical model of this paper to a modern implementation. What is your view about that comparison? e.g. how do you think about the following possibilities:
There is no material weakness in the security proof.
A material weakness is already known.
An interested layperson could find a material weakness with moderate effort.
An expert could find a material weakness with significant effort.
My guess would be that probably we’re in world 2, and if not that it’s probably because no one cares that much (e.g. because it’s obvious that there will be some material weakness and the standards of the field are such that it’s not publishable unless it actually comes with an attack) and we are in world 3.
(On a quick skim, and from the author’s language when describing the model, my guess is that material weaknesses of the model are more or less obvious and that the authors are aware of potential attacks not covered by their model.)
I’m still curious for your view on the crypto examples you cited. My current understanding is that people do not expect the security proofs to rule out all possible attacks (a situation I can sympathize with since I’ve written multiple proofs that rule out large classes of attacks without attempting to cover all possible attacks), so I’m interested in whether (i) you disagree with that and believe that serious onlookers have had the expectation that proofs are comprehensive, (ii) you agree but feel it would be impractical to give a correct proof and this is a testament to the difficulty of proving things, (iii) you feel it would be possible but prohibitively expensive, and are expressing a quantitative point about the cost of alignment analyses being impractical, (iv) you feel that the crypto case would be practical but the AI case is likely to be much harder and just want to make a directionally analogous update.
I still feel like more of the action is in my skepticism about the (alignment analysis) <--> (security analysis) analogy, but I could still get some update out of the analogy if the crypto situation is thornier than I currently believe.
I think that post is basically talking about the same kinds of hard cases as in Towards Formalizing Universality 1.5 years earlier (in section IV), so it’s intended to be more about clarification/exposition than changing views.
See the thread with Rohin above for some rough history.
I’m not sure.It’s possible I would become more pessimistic if I walked through concrete cases of people’s analyses being wrong in subtle and surprising ways.
My experience with practical systems is that it is usually easy for theorists to describe hypothetical breaks for the security model, and the issue is mostly one of prioritization (since people normally don’t care too much about security). For example, my strong expectation would be that people had described hypothetical attacks on any of the systems discussed in the article you linked prior to their implementation, at least if they had ever been subject to formal scrutiny. The failures are just quite far away from the levels of paranoia that I’ve seen people on the theory side exhibit when they are trying to think of attacks.
I would also expect that e.g. if you were to describe almost any existing practical system with purported provable security, it would be straightforward for a layperson with theoretical background (e.g. me) to describe possible attacks that are not precluded by the security proof, and that it wouldn’t even take that long. It sounds like a fun game.
Another possible divergence is that I’m less convinced by the analogy, since alignment seems more about avoiding the introduction of adversarial consequentialists and it’s not clear if that game behaves in the same way. I’m not sure if that’s more or less important than the prior point.
I would want to do a lot of work before deploying an algorithm in any context where a failure would be catastrophic (though “before letting it become potentially superintelligent” kind of suggests a development model I’m not on board with).
That would ideally involve theoretical analysis from a lot of angles, e.g. proofs of key properties that are amenable to proof, demonstrations of how the system could plausibly fail if we were wrong about key claims or if we relax assumptions, and so on.
It would also involve good empirical characterization, including things like running on red team inputs, or changing the training procedure in ways that seem as bad as possible while still preserving our alignment arguments, and performing extensive evals under those more pessimistic conditions. It would involve validating key claims individually, and empirically testing other claims that are established by structurally similar arguments. It would involve characterizing scaling behavior where applicable and understanding it as well as we can (along with typical levels of variability and plausible stories about deviations from trend).
I’m not exactly sure what you are asking. It seems like we’ll do what we can on all the fronts and prioritize them as well as we can. Do you mean, what else can we say today about what methodologies we’d use? Or under what conditions would I pivot to spending down my political capital to delay deployment? Or something else?