But what stops a blue-cloud model from transitioning into a red-cloud model if the blue-cloud model is an AGI like the one hinted at on your slides (self-aware, goal-directed, highly competent)?
GunZoR
Karma: 7
If it’s impossible in principle to know whether any AI really has qualia, then what’s wrong with simply using the Turing test as an ultimate ethical safeguard? We don’t know how consciousness works, and possibly we won’t ever (e.g., mysterianism might obtain). But certainly we will soon create an AI that passes the Turing test. So seemingly we have good ethical reasons just to assume that any agent that passes the Turing test is sentient — this blanket assumption, even if often unwarranted from the aspect of eternity, will check our egos and thereby help prevent ethical catastrophe. And I don’t see that any more sophisticated ethical reasoning around AI sentience is or ever will be needed. Then the resolution of what’s really happening inside the AI will simply continually increase over time; and, without worry, we’ll be able to look back and perhaps see where we were right and wrong. Meanwhile, we can focus less on ethics and more on alignment.
But what is stopping any of those “general, agentic learning systems” in the class “aligned to human values” from going meta — at any time — about its values and picking different values to operate with? Is the hope to align the agent and then constantly monitor it to prevent deviancy? If so, why wouldn’t preventing deviancy by monitoring be practically impossible, given that we’re dealing with an agent that will supposedly be able to out-calculate us at every step?
Mental Impoverishment
We should be trying to create mentally impoverished AGI, not profoundly knowledgeable AGI — no matter how difficult this is relative to the current approach of starting by feeding our AIs a profound amount of knowledge.
If a healthy five-year-old[1] has GI and qualia and can pass the Turing test, then a necessary condition of GI and qualia and passing the Turing test isn’t profound knowledge. A healthy five-year-old does have GI and qualia and can pass the Turing test. So a necessary condition of GI and qualia and passing the Turing test isn’t profound knowledge.
If GI and qualia and the ability to pass the Turing test don’t require profound knowledge in order to arise in a biological system, then GI and qualia and the ability to pass the Turing test don’t require profound knowledge in order to arise in a synthetic material [this premise seems to follow from the plausible assumption of substrate-independence]. GI and qualia and the ability to pass the Turing test don’t require profound knowledge in order to arise in a biological system. So GI and qualia and the ability to pass the Turing test don’t require profound knowledge in order to arise in a synthetic material.
A GI with qualia and the ability to pass the Turing test which arises in a synthetic material and doesn’t have profound knowledge is much less dangerous than a GI with qualia and the ability to pass the Turing test which arises in a synthetic material and does have profound knowledge. (This also seems to be true of [] a GI without qualia and the inability to pass the Turing test which arises in a synthetic material and does not have profound knowledge; and of [] a GI without qualia and the ability to pass the Turing test which arises in a synthetic material and doesn’t have profound knowledge.)
So we ought to be trying to create either (A) a synthetic-housed GI that can pass the Turing test without qualia and without profound knowledge, or (B) a synthetic-housed GI that can pass the Turing test with qualia and without profound knowledge.
Either of these paths — the creation of (A) or (B) — is preferable to our current path, no matter how long they delay the arrival of AGI. In other words, it is preferable that we create AGI in years than that we create AGI in if creating AGI in means humanity’s loss of dominance or its destruction.
- ^
My arguable assumption is that what makes a five-year-old generally less dangerous than, say, an adult Einstein is a relatively profound lack of knowledge (even physical know-how seems to be a form of knowledge). All other things being equal, if a five-year-old has the knowledge of how to create a pipe bomb, he is just as dangerous as an adult Einstein with the same knowledge, if “knowledge” means something like “accessible complete understanding of .”
- ^
I have got the faint suspicion that a tone of passive-aggressive condescension isn’t optimal here …