I cannot fathom how this is any more than a distraction from the hardline reality that when human beings gain the ability to manufacture “agent AI”, we WILL.
Any number of companies and/or individuals can ethically choose to focus on “tool AI” rather than “agent AI”, but that will never erase the inevitable human need to create that which it believes and/or knows it can create.
In simple terms, SI’s viewpoint (as I understand it) is that “agent AI’s” are inevitable.… some group or individual somewhere at some point WILL produce the phenomenon, if for no other reason than because it is human nature to look behind the curtain no matter what the inherent risks may be. History has no shortage of proof in support of this truth.
SI asserts that (again, as I understand it) it is imperative for someone to at least attempt to create a friendly “agent AI” FIRST, so there is at least a chance that human interests will be part of the evolving equation… an equation that could potentially change too quickly for humans to assume there will be time for testing or second chances.
I am not saying I agree with SI’s stance, but I don’t see how an argument that SI should spend time, money and energy on a possible alternative to “agent AI” is even relevant when the point is explicitly that it doesn’t matter how many alternatives there are nor how much more safe they may be to humans; “agent AI” WILL happen at some point in the future and its impact should be addressed, even if our attempts at addressing those impacts are ultimately futile due to unforseen developments.
Try applying Karnofsky’s style of argument above to the creation of the atomic bomb. Using the logic of this argument in a pre-atomic world, one would simply say, “It will be fine so long as we all agree NOT to go there. Let’s work on something similar, but with less destructive force,” and expecting this to stop the scientists of the world from proceeding to produce an atomic bomb. Once the human mind becomes aware of the possibility of something that was once considered beyond comprehension, it will never rest until it has been achieved.
No, I”m sure it is just my lack of knowledge regarding philosophy and the associated methods of discussing it. I never actually believed that the author was trying to convince me that death was not bad, but (as I stated above) playing devil’s advocate in order to explore ideas and challenge the reader. I simply wouldn’t know enough about it to name it the “Socratic method”. My bad.