Who knows… The OP makes a strong argument that the AIs will inherit a lot of our values, but we can’t be sure how those values will be modified in the long run (of course, the same would be true about an AI-free civilization of humans, we don’t know how that human civilization would modify our values in the long run).
The problem of keeping a particularly important subset of values invariant in the long run is a rather non-trivial problem. I have spent quite a bit of time trying to contribute to its solution, and as a result of those efforts I do think that it can be solved (within reason), but whether a set of methods capable of solving it will actually be adopted is not clear. (When one ponders the problem of human survival and flourishing, it soon becomes apparent that the ability to keep some subset of values invariant in the long term is crucial for that as well, so I hope we’ll see a bit more focus on that from the community focusing on AI existential safety.)
facilitating the ability to “think from a variety of viewpoints”
It can be facilitated in other ways. Why do you think AIs would choose this exact way?
I think AIs will choose all available ways which are capable of improving the “coverage”.
I expect them to be quite diligent in exercising all opportunities to improve the quality of their thinking.
Better for what value system?
It can be facilitated in other ways. Why do you think AIs would choose this exact way?
Who knows… The OP makes a strong argument that the AIs will inherit a lot of our values, but we can’t be sure how those values will be modified in the long run (of course, the same would be true about an AI-free civilization of humans, we don’t know how that human civilization would modify our values in the long run).
The problem of keeping a particularly important subset of values invariant in the long run is a rather non-trivial problem. I have spent quite a bit of time trying to contribute to its solution, and as a result of those efforts I do think that it can be solved (within reason), but whether a set of methods capable of solving it will actually be adopted is not clear. (When one ponders the problem of human survival and flourishing, it soon becomes apparent that the ability to keep some subset of values invariant in the long term is crucial for that as well, so I hope we’ll see a bit more focus on that from the community focusing on AI existential safety.)
I think AIs will choose all available ways which are capable of improving the “coverage”.
I expect them to be quite diligent in exercising all opportunities to improve the quality of their thinking.