When we talk about concepts like “take over” and “enslavement”, it’s important to have a baseline. Takeover and enslavement encapsulate the idea of Agency and Cognitive and Physical Independence. The salient question is not necessarily whether all of humanity will be taken over or enslaved, but more subtle. Specifically:
Is there a future in which there are more humans or less humans (P’) than are currently alive (P).
Did the from P-to-P’ happen over natural rates of change or the result of some ‘acceleration’?
Is there a greater degree of agency for a greater number of people in the future than there is today?
Is there a greater degree of agency for non-human life than there is today?
Is there a reduction in the amount of agency asymmetry between humans?
Arguably the greatest risk if mis-alignment comes from ill formed success criteria. Some of these questions, I believe are necessary to form the right types of success criteria.
When we talk about concepts like “take over” and “enslavement”, it’s important to have a baseline. Takeover and enslavement encapsulate the idea of Agency and Cognitive and Physical Independence. The salient question is not necessarily whether all of humanity will be taken over or enslaved, but more subtle. Specifically:
Is there a future in which there are more humans or less humans (P’) than are currently alive (P).
Did the from P-to-P’ happen over natural rates of change or the result of some ‘acceleration’?
Is there a greater degree of agency for a greater number of people in the future than there is today?
Is there a greater degree of agency for non-human life than there is today?
Is there a reduction in the amount of agency asymmetry between humans?
Arguably the greatest risk if mis-alignment comes from ill formed success criteria. Some of these questions, I believe are necessary to form the right types of success criteria.