I feel a bit surprised by how much you dislike Section 3. I agree that it does not address ‘the strongest counterarguments and automated-alignment plans that haven’t been written down publicly’; this is a weakness but seems too demanding given what’s public.
I particularly like the analogy to alchemy presented in Chapter 11. I think it is basically correct (or as correct as analogies get) that the state of AI alignment research is incredibly poor and the field is in its early stages where we have no principled understanding of anything (my belief here is based on reading or skimming basically every AI safety paper in 2024). The next part of the argument is like “we’re not going to be able to get from the present state of alchemy to a ‘mature scientific field that doesn’t screw up certain crucial problems on the first try’ in time”. That is, 1: the field is currently very early stages without principled understanding, 2: we’re not going to be able to get from where we are now to a sufficient level by the time we need.
My understanding is that your disagreement is with 2? You think that earlier AIs are going to be able to dramatically speed up alignment research (and by using control methods we can get more alignment research out of better AIs, for some intermediate capability levels), getting us to the principled, doesn’t-mess-up-the-first-try-on-any-critical-problem place before ASI.
Leaning into the analogy, I would describe what I view as your position as “with AI assistance, we’re going to go from alchemy to first-shot-moon-landing in ~3 years of wall clock time”. I think it’s correct for people to think this position is very crazy at first glance. I’ve thought about it some and think it’s only moderately crazy. I am glad that Ryan is working on better plans here (and excited to potentially update my beliefs, as I did when you all put out various pieces about AI Control), but I think the correct approach for people hearing about this plan is to be very worried about this plan.
I really liked Section 3, especially Ch 11, because it makes this (IMO) true and important point about the state of the AI alignment field. I think this argument stands on its own as a reason to have an AI moratorium, even absent the particular arguments about alignment difficulty in Section 1. Meanwhile, it sounds like you don’t like this section because, to put it disingenuously, “they don’t engage with my favorite automating-alignment plan that tries to get us from alchemy to first-shot-moon-landing in ~3 years of wall clock time and that hasn’t been written down anywhere”.
Also, if you happen to disagree strongly with the analogy to alchemy or 1 above (e.g., think it’s an incorrect frame), that would be interesting to hear! Perhaps the disagreement is in how hard alignment problems will be in the development of ASI; for example, if the alchemists merely had to fly a blimp first try, rather than land a rocket on the moon? Perhaps you don’t expect there to be any significant discontinuities and this whole “first try” claim is wrong and we’ll never need a principled understanding?
I found this post and your review to be quite thoughtful overall!
I feel a bit surprised by how much you dislike Section 3. I agree that it does not address ‘the strongest counterarguments and automated-alignment plans that haven’t been written down publicly’; this is a weakness but seems too demanding given what’s public.
I particularly like the analogy to alchemy presented in Chapter 11. I think it is basically correct (or as correct as analogies get) that the state of AI alignment research is incredibly poor and the field is in its early stages where we have no principled understanding of anything (my belief here is based on reading or skimming basically every AI safety paper in 2024). The next part of the argument is like “we’re not going to be able to get from the present state of alchemy to a ‘mature scientific field that doesn’t screw up certain crucial problems on the first try’ in time”. That is, 1: the field is currently very early stages without principled understanding, 2: we’re not going to be able to get from where we are now to a sufficient level by the time we need.
My understanding is that your disagreement is with 2? You think that earlier AIs are going to be able to dramatically speed up alignment research (and by using control methods we can get more alignment research out of better AIs, for some intermediate capability levels), getting us to the principled, doesn’t-mess-up-the-first-try-on-any-critical-problem place before ASI.
Leaning into the analogy, I would describe what I view as your position as “with AI assistance, we’re going to go from alchemy to first-shot-moon-landing in ~3 years of wall clock time”. I think it’s correct for people to think this position is very crazy at first glance. I’ve thought about it some and think it’s only moderately crazy. I am glad that Ryan is working on better plans here (and excited to potentially update my beliefs, as I did when you all put out various pieces about AI Control), but I think the correct approach for people hearing about this plan is to be very worried about this plan.
I really liked Section 3, especially Ch 11, because it makes this (IMO) true and important point about the state of the AI alignment field. I think this argument stands on its own as a reason to have an AI moratorium, even absent the particular arguments about alignment difficulty in Section 1. Meanwhile, it sounds like you don’t like this section because, to put it disingenuously, “they don’t engage with my favorite automating-alignment plan that tries to get us from alchemy to first-shot-moon-landing in ~3 years of wall clock time and that hasn’t been written down anywhere”.
Also, if you happen to disagree strongly with the analogy to alchemy or 1 above (e.g., think it’s an incorrect frame), that would be interesting to hear! Perhaps the disagreement is in how hard alignment problems will be in the development of ASI; for example, if the alchemists merely had to fly a blimp first try, rather than land a rocket on the moon? Perhaps you don’t expect there to be any significant discontinuities and this whole “first try” claim is wrong and we’ll never need a principled understanding?
I found this post and your review to be quite thoughtful overall!