I cannot fathom how this is any more than a distraction from the hardline reality that when human beings gain the ability to manufacture “agent AI”, we WILL.
Any number of companies and/or individuals can ethically choose to focus on “tool AI” rather than “agent AI”, but that will never erase the inevitable human need to create that which it believes and/or knows it can create.
In simple terms, SI’s viewpoint (as I understand it) is that “agent AI’s” are inevitable.… some group or individual somewhere at some point WILL produce the phenomenon, if for no other reason than because it is human nature to look behind the curtain no matter what the inherent risks may be. History has no shortage of proof in support of this truth.
SI asserts that (again, as I understand it) it is imperative for someone to at least attempt to create a friendly “agent AI” FIRST, so there is at least a chance that human interests will be part of the evolving equation… an equation that could potentially change too quickly for humans to assume there will be time for testing or second chances.
I am not saying I agree with SI’s stance, but I don’t see how an argument that SI should spend time, money and energy on a possible alternative to “agent AI” is even relevant when the point is explicitly that it doesn’t matter how many alternatives there are nor how much more safe they may be to humans; “agent AI” WILL happen at some point in the future and its impact should be addressed, even if our attempts at addressing those impacts are ultimately futile due to unforseen developments.
Try applying Karnofsky’s style of argument above to the creation of the atomic bomb. Using the logic of this argument in a pre-atomic world, one would simply say, “It will be fine so long as we all agree NOT to go there. Let’s work on something similar, but with less destructive force,” and expecting this to stop the scientists of the world from proceeding to produce an atomic bomb. Once the human mind becomes aware of the possibility of something that was once considered beyond comprehension, it will never rest until it has been achieved.
This logic assumes that a beyond human intelligence in a redesigned world would still find inherent value in free will. Isn’t it possible that such an intelligence would move beyond the need to experience pain in order to comprehend the value of pleasure?
According to the bible, god created different aspects of the world across six days and after each creation he “saw that it was good”. Yet nothing ELSE existed. If there had never been a “world” before, and evil had not yet been unleashed, by what method was this god able to measure that his creation was good? One must assume that god’s superior intelligence simply KNEW it to be good and had no need to measure it against something “bad” in order to know it. Couldn’t the eventual result of AI be the attainment of the same ability… the ability to KNOW pleasure without the existence of its opposite?
Isn’t the hope (or should I say fun?) of considering the potential of AI that such a vast intelligence would move life BEYOND the anchors to which we now find ourselves locked? If AI is simply going to be filled with the same needs and methods of measuring “happiness” as we currently deal with, what is the point of hoping for it at all?
This is a bit of an afterthought, but even at our current level of intelligence, humans have no way of knowing if we would value pleasure if pain did not exist. Pain does now and has always existed. “Evil” (or what we perceive as evil) has existed since the dawn of recorded human existence. How can we assume that we are not already capable of recognizing pleasure as pleasure and good as good without their opposites to compare them to? We have never had the opportunity to try.