I think there’s a nontrivial probability that continual learning (automated adaptation), if done right (in the reckless sense of not engaging with an AGI Pause), could make early AGIs into people on a distribution of values that heavily overlaps that of humans. This doesn’t solve most problems, but some aspects of alien nature might go away more thoroughly than usually expected.
A crux for this is probably that I consider humans as already occupying a wider variety of values-on-reflection than usually expected, in a way that’s largely untethered from biologically encoded psychological adaptations, and it’s primarily society and culture that create the impression (and on some level the reality) of coherence and shared values. If AGIs merely slot into this framework, and manage to establish an ASI Pause (provided ASI-grade alignment really is hard), it’s likely that everyone literally dying is not the outcome. Though AGIs will still be taking almost all of the Future for the normal selfish reasons (resulting in permanent disempowerment for the future of humanity).
I think there’s a nontrivial probability that continual learning (automated adaptation), if done right (in the reckless sense of not engaging with an AGI Pause), could make early AGIs into people on a distribution of values that heavily overlaps that of humans. This doesn’t solve most problems, but some aspects of alien nature might go away more thoroughly than usually expected.
A crux for this is probably that I consider humans as already occupying a wider variety of values-on-reflection than usually expected, in a way that’s largely untethered from biologically encoded psychological adaptations, and it’s primarily society and culture that create the impression (and on some level the reality) of coherence and shared values. If AGIs merely slot into this framework, and manage to establish an ASI Pause (provided ASI-grade alignment really is hard), it’s likely that everyone literally dying is not the outcome. Though AGIs will still be taking almost all of the Future for the normal selfish reasons (resulting in permanent disempowerment for the future of humanity).