You set the article up with an argument along the lines of: “There is no technical work that we can do right now to increase the probability that AGI goes well.”
It seems to me that the only possible counterargument is “Actually, there is technical work that we can do right now to increase the probability that AGI goes well”.
I say that this is only possible counterargument, because how else can you possibly support doing technical AGI safety work? Having a bunch of people doing knowably-useless work—nobody wants that! It won’t help anything!
Anyway, I think the evidence for “Actually, there is technical work that we can do right now to increase the probability that AGI goes well” is strong.
For example, we have at least three ways to reason about what AGI will look like—ML systems of today (“Prosaic AGI”), human brains [yay!], and “things that are fundamentally required for a system to qualify as AGI” (e.g. what is the fundamental nature of rational agency, what does “planning” mean, etc.). We also have various more specific existing lines of technical research, and we can try to evaluate whether they might be useful. We also have the possibility of finding new directions for technical research, even if all the existing ones are lousy.
Many of your counterarguments seem to be slightly missing the point to me—not directly addressing the argument at the top. Or maybe I’m misunderstanding.
I guess the main reason my arguments are not addressing the argument at the top is that I interpreted Aaronson’s and Garfinkel’s arguments as “It’s highly uncertain whether any of the technical work we can do today will be useful” rather than as “There is no technical work that we can do right now to increase the probability that AGI goes well.” I think that it’s possible to respond to the former with “Even if it is so and this work really does have a high chance of being useless, there are many good reasons to nevertheless do it,” while assuming the latter inevitably leads to the conclusion that one should do something else instead of this knowably-useless work.
My aim with this post was to take an agnostic standpoint towards whether that former argument is true and to argue that even if it is, there are still good reasons to work on AI safety. I chose this framing because I believe that for people new to the field who don’t yet know enough about the field to make good guesses about how likely it is that AGI will be similar to ML systems of today or to human brains, it’s useful to think about whether it’s worth working on AI safety even if the chance that we’ll build prosaic or brain-like AGI turns out to be low.
That being said, I could have definitely done a better job writing the post—for example by laying out the claim I’m arguing against more clearly at the start and by connecting argument 4 more directly to the argument that there’s a significant chance we’ll build a prosaic or brain-like AGI. It might also be that the quotes by Aaronson and Garfinkel convey the argument you thought I’m arguing against rather than what I interpreted them to convey. Thank you for the feedback and for helping me realize the post might have these problems!
You set the article up with an argument along the lines of: “There is no technical work that we can do right now to increase the probability that AGI goes well.”
It seems to me that the only possible counterargument is “Actually, there is technical work that we can do right now to increase the probability that AGI goes well”.
I say that this is only possible counterargument, because how else can you possibly support doing technical AGI safety work? Having a bunch of people doing knowably-useless work—nobody wants that! It won’t help anything!
Anyway, I think the evidence for “Actually, there is technical work that we can do right now to increase the probability that AGI goes well” is strong.
For example, we have at least three ways to reason about what AGI will look like—ML systems of today (“Prosaic AGI”), human brains [yay!], and “things that are fundamentally required for a system to qualify as AGI” (e.g. what is the fundamental nature of rational agency, what does “planning” mean, etc.). We also have various more specific existing lines of technical research, and we can try to evaluate whether they might be useful. We also have the possibility of finding new directions for technical research, even if all the existing ones are lousy.
Many of your counterarguments seem to be slightly missing the point to me—not directly addressing the argument at the top. Or maybe I’m misunderstanding.
I guess the main reason my arguments are not addressing the argument at the top is that I interpreted Aaronson’s and Garfinkel’s arguments as “It’s highly uncertain whether any of the technical work we can do today will be useful” rather than as “There is no technical work that we can do right now to increase the probability that AGI goes well.” I think that it’s possible to respond to the former with “Even if it is so and this work really does have a high chance of being useless, there are many good reasons to nevertheless do it,” while assuming the latter inevitably leads to the conclusion that one should do something else instead of this knowably-useless work.
My aim with this post was to take an agnostic standpoint towards whether that former argument is true and to argue that even if it is, there are still good reasons to work on AI safety. I chose this framing because I believe that for people new to the field who don’t yet know enough about the field to make good guesses about how likely it is that AGI will be similar to ML systems of today or to human brains, it’s useful to think about whether it’s worth working on AI safety even if the chance that we’ll build prosaic or brain-like AGI turns out to be low.
That being said, I could have definitely done a better job writing the post—for example by laying out the claim I’m arguing against more clearly at the start and by connecting argument 4 more directly to the argument that there’s a significant chance we’ll build a prosaic or brain-like AGI. It might also be that the quotes by Aaronson and Garfinkel convey the argument you thought I’m arguing against rather than what I interpreted them to convey. Thank you for the feedback and for helping me realize the post might have these problems!