It seems that consciousness is one of the essential things necessary to characterize the moral status of an agent. It seems that we have very little chance to solve the AI safety problem. We don’t know if the first AGI will be conscious.
Since we are in a certain sense doomed, the future consists of the following 2 cases:
Either the first AGI is not sentient and never becomes sentient.
Or the first AGI acquires sentience at a given moment.
Either the valence of this AGI is positive
Or the valence of this AGI is negative
Either the valence is mixed
Or the AGI reprograms itself to experience a positive valence (instrumental convergence if its well-being is included in the utility function?)
I have no idea which scenario is the most desirable. The uncertainty is multiplied by considering the AGI’s superior capabilities, and its ability to multiply. So perhaps we could see it as a utility monster. Therefore, the following questions seem very important:
(1) What do you think is the most likely scenario and (2) the most desirable scenario?
(3|2) Conditional on the scenario that seems most desirable to you, is there a way to steer the future in that direction?
[Question] Is it desirable for the first AGI to be conscious?
It seems that consciousness is one of the essential things necessary to characterize the moral status of an agent.
It seems that we have very little chance to solve the AI safety problem.
We don’t know if the first AGI will be conscious.
Since we are in a certain sense doomed, the future consists of the following 2 cases:
Either the first AGI is not sentient and never becomes sentient.
Or the first AGI acquires sentience at a given moment.
Either the valence of this AGI is positive
Or the valence of this AGI is negative
Either the valence is mixed
Or the AGI reprograms itself to experience a positive valence (instrumental convergence if its well-being is included in the utility function?)
I have no idea which scenario is the most desirable. The uncertainty is multiplied by considering the AGI’s superior capabilities, and its ability to multiply. So perhaps we could see it as a utility monster. Therefore, the following questions seem very important:
(1) What do you think is the most likely scenario and (2) the most desirable scenario?
(3|2) Conditional on the scenario that seems most desirable to you, is there a way to steer the future in that direction?