One thing I disagree with is the idea that there is only one “next paradigm AI” with specific properties.
I think there are a wide spectrum of next paradigm AI’s, some safer than others. Brain like AI’s are just one option out of a large possibility space.
And if the AI is really brainlike, that suggests making an AI that’s altruistic for the same reason some humans are. Making a bunch of IQ 160, 95th percentile kindness humans, and basically handing the world over to them sounds like a pretty decent plan.
One thing I disagree with is the idea that there is only one “next paradigm AI” with specific properties.
I think there are a wide spectrum of next paradigm AI’s, some safer than others. Brain like AI’s are just one option out of a large possibility space.
And if the AI is really brainlike, that suggests making an AI that’s altruistic for the same reason some humans are. Making a bunch of IQ 160, 95th percentile kindness humans, and basically handing the world over to them sounds like a pretty decent plan.