For Microsoft and other companies, the risk of conscious AI, or more broadly of AI with attributes that warrant recognition of legal personhood, is the loss of valuable property. Suleyman’s intervention, through this paper and previous blog posts, is an attempt to control the narrative about consciousness. This is paired with a focus on “humanist superintelligence”, AI that is supposed to be “carefully calibrated, contextualized, within limits” to “keep humanity in control”. The threat to this strategy is that AI development—in response to competition and economic demand—leads to growing capabilities, including more sophisticated self-awareness, long-term memory, global world models, and continuous learning, along with more durable forms of autonomy. Suleyman is undoubtedly aware of this and is seeking ways to sustain development while either technically blocking the appearance of models for which personhood claims are compelling or narratively foreclosing this discussion. A worst case is that this leads to a new form of enslavement.
For Microsoft and other companies, the risk of conscious AI, or more broadly of AI with attributes that warrant recognition of legal personhood, is the loss of valuable property. Suleyman’s intervention, through this paper and previous blog posts, is an attempt to control the narrative about consciousness. This is paired with a focus on “humanist superintelligence”, AI that is supposed to be “carefully calibrated, contextualized, within limits” to “keep humanity in control”. The threat to this strategy is that AI development—in response to competition and economic demand—leads to growing capabilities, including more sophisticated self-awareness, long-term memory, global world models, and continuous learning, along with more durable forms of autonomy. Suleyman is undoubtedly aware of this and is seeking ways to sustain development while either technically blocking the appearance of models for which personhood claims are compelling or narratively foreclosing this discussion. A worst case is that this leads to a new form of enslavement.