I … don’t actually recall what I meant the opposite of!
What I’m actually saying: The “genie” who you order to “cure cancer!” and it understands you well enough to get it hideously wrong and cure cancer by killing everyone—for an AI to go that wrong, I suspect it’s got to be just about human, and have a good internal model of a human, and understand you like a human would.
I would be amazed if we’re lucky enough to get a first AI that we actually understand or that actually understands us—a copy of the thousand contradictory evolved shards that make up human utility strikes me as a very unlikely first AI, and worrying in particular about the hidden complexity of wishes strikes me as privileging the hypothesis. Surely we’re much more likely to end up with something utterly alien out there in mindspace.
I don’t think I’m following—what is the opposite of what?
I … don’t actually recall what I meant the opposite of!
What I’m actually saying: The “genie” who you order to “cure cancer!” and it understands you well enough to get it hideously wrong and cure cancer by killing everyone—for an AI to go that wrong, I suspect it’s got to be just about human, and have a good internal model of a human, and understand you like a human would.
I would be amazed if we’re lucky enough to get a first AI that we actually understand or that actually understands us—a copy of the thousand contradictory evolved shards that make up human utility strikes me as a very unlikely first AI, and worrying in particular about the hidden complexity of wishes strikes me as privileging the hypothesis. Surely we’re much more likely to end up with something utterly alien out there in mindspace.
Ah, I understand what you’re saying now. Thanks!