I guess the main reason my arguments are not addressing the argument at the top is that I interpreted Aaronson’s and Garfinkel’s arguments as “It’s highly uncertain whether any of the technical work we can do today will be useful” rather than as “There is no technical work that we can do right now to increase the probability that AGI goes well.” I think that it’s possible to respond to the former with “Even if it is so and this work really does have a high chance of being useless, there are many good reasons to nevertheless do it,” while assuming the latter inevitably leads to the conclusion that one should do something else instead of this knowably-useless work.
My aim with this post was to take an agnostic standpoint towards whether that former argument is true and to argue that even if it is, there are still good reasons to work on AI safety. I chose this framing because I believe that for people new to the field who don’t yet know enough about the field to make good guesses about how likely it is that AGI will be similar to ML systems of today or to human brains, it’s useful to think about whether it’s worth working on AI safety even if the chance that we’ll build prosaic or brain-like AGI turns out to be low.
That being said, I could have definitely done a better job writing the post—for example by laying out the claim I’m arguing against more clearly at the start and by connecting argument 4 more directly to the argument that there’s a significant chance we’ll build a prosaic or brain-like AGI. It might also be that the quotes by Aaronson and Garfinkel convey the argument you thought I’m arguing against rather than what I interpreted them to convey. Thank you for the feedback and for helping me realize the post might have these problems!
Thanks, that definitely seems like a great way to gather these ideas together!