I mean, analogies don’t have to be similar in all respects to be useful explanations, just in the few respects that you’re using the analogy for. OP isn’t arguing that AI alignment is important because rocket alignment is important, it’s only using the analogy to describe the type of work that it thinks needs to be done to align AGI—which I’m guessing has been difficult to describe before writing this post. Arguments that AGI needs to be built right the first time have been discussed elsewhere, and you’re right that this post doesn’t make that arg.
(On this side-topic of whether AGI needs to be build precisely right first time, and counter to your point that we-always-get-stuff-wrong-a-bunch-at-first-and-that’s-fine, I liked Max Tegmark’s story of how we’re building technologies that increasingly have less affordance for error—fire, nukes, AGI—some of these having a few mistakes was of small damage, then of big damage, and in principle we may hit tech where initial mistakes are existential in nature. I think there are some sane args that make AGI seems like a plausible instance of this.
My point, clearly not well expressed, is that the main issue why the AI alignment has to be figured out in advance is not even mentioned in the OP’s dialogue:
We think the most important thing to do next is to advance our understanding of rocket trajectories until we have a better, deeper understanding of what we’ve started calling the “rocket alignment problem” There are other safety problems, but this rocket alignment problem will probably take the most total time to work on, so it’s the most urgent.
… why? So what if this problem remains after the other problems are solved and the rockets are flying every which way? I have tried to answer that, since Eliezer hasn’t in this post, despite this being the main impetus of MIRI’s work.
I feel like the post is trying to convince the reader that AI alignment needs to be solved AT ALL. You can worry about arguing when it needs to be solved after the other person in convinced there is a problem to solve in the first place.
I mean, analogies don’t have to be similar in all respects to be useful explanations, just in the few respects that you’re using the analogy for. OP isn’t arguing that AI alignment is important because rocket alignment is important, it’s only using the analogy to describe the type of work that it thinks needs to be done to align AGI—which I’m guessing has been difficult to describe before writing this post. Arguments that AGI needs to be built right the first time have been discussed elsewhere, and you’re right that this post doesn’t make that arg.
(On this side-topic of whether AGI needs to be build precisely right first time, and counter to your point that we-always-get-stuff-wrong-a-bunch-at-first-and-that’s-fine, I liked Max Tegmark’s story of how we’re building technologies that increasingly have less affordance for error—fire, nukes, AGI—some of these having a few mistakes was of small damage, then of big damage, and in principle we may hit tech where initial mistakes are existential in nature. I think there are some sane args that make AGI seems like a plausible instance of this.
For discussion of the AI details I’d point elsewhere, to things like Gwern on “Why Tool AIs Want to be Agent AIs”, Paul Christiano discussing arguments for fast-takeoff speeds, the paper Intelligence Explosion Microeconomics, and of course Bostrom’s book.)
(edited a few lines to be clearer/shorter)
My point, clearly not well expressed, is that the main issue why the AI alignment has to be figured out in advance is not even mentioned in the OP’s dialogue:
… why? So what if this problem remains after the other problems are solved and the rockets are flying every which way? I have tried to answer that, since Eliezer hasn’t in this post, despite this being the main impetus of MIRI’s work.
I feel like the post is trying to convince the reader that AI alignment needs to be solved AT ALL. You can worry about arguing when it needs to be solved after the other person in convinced there is a problem to solve in the first place.