However, it is likely that some AI systems will have access to human consciousness via “merge” setups and will have options to experience what some of the humans experience.
If all of the AI systems somehow end up not valuing that, presumably that would mean they end up having something even better?
(By default, if we end up having sentient AI individuals at all, I would expect that many of them would chose hedonic exploration of a great variety of subjective realms. Exploring a variety of subjective realms seems to provide plenty of “immediate terminal value” for us; it also seems to have cognitive value for any entities, facilitating the ability to “think from a variety of viewpoints”. We can’t be certain about all this, but this does seem likely given that the AIs will be very much aware of these possibilities.)
Who knows… The OP makes a strong argument that the AIs will inherit a lot of our values, but we can’t be sure how those values will be modified in the long run (of course, the same would be true about an AI-free civilization of humans, we don’t know how that human civilization would modify our values in the long run).
The problem of keeping a particularly important subset of values invariant in the long run is a rather non-trivial problem. I have spent quite a bit of time trying to contribute to its solution, and as a result of those efforts I do think that it can be solved (within reason), but whether a set of methods capable of solving it will actually be adopted is not clear. (When one ponders the problem of human survival and flourishing, it soon becomes apparent that the ability to keep some subset of values invariant in the long term is crucial for that as well, so I hope we’ll see a bit more focus on that from the community focusing on AI existential safety.)
facilitating the ability to “think from a variety of viewpoints”
It can be facilitated in other ways. Why do you think AIs would choose this exact way?
I think AIs will choose all available ways which are capable of improving the “coverage”.
I expect them to be quite diligent in exercising all opportunities to improve the quality of their thinking.
That’s true.
However, it is likely that some AI systems will have access to human consciousness via “merge” setups and will have options to experience what some of the humans experience.
If all of the AI systems somehow end up not valuing that, presumably that would mean they end up having something even better?
(By default, if we end up having sentient AI individuals at all, I would expect that many of them would chose hedonic exploration of a great variety of subjective realms. Exploring a variety of subjective realms seems to provide plenty of “immediate terminal value” for us; it also seems to have cognitive value for any entities, facilitating the ability to “think from a variety of viewpoints”. We can’t be certain about all this, but this does seem likely given that the AIs will be very much aware of these possibilities.)
Better for what value system?
It can be facilitated in other ways. Why do you think AIs would choose this exact way?
Who knows… The OP makes a strong argument that the AIs will inherit a lot of our values, but we can’t be sure how those values will be modified in the long run (of course, the same would be true about an AI-free civilization of humans, we don’t know how that human civilization would modify our values in the long run).
The problem of keeping a particularly important subset of values invariant in the long run is a rather non-trivial problem. I have spent quite a bit of time trying to contribute to its solution, and as a result of those efforts I do think that it can be solved (within reason), but whether a set of methods capable of solving it will actually be adopted is not clear. (When one ponders the problem of human survival and flourishing, it soon becomes apparent that the ability to keep some subset of values invariant in the long term is crucial for that as well, so I hope we’ll see a bit more focus on that from the community focusing on AI existential safety.)
I think AIs will choose all available ways which are capable of improving the “coverage”.
I expect them to be quite diligent in exercising all opportunities to improve the quality of their thinking.