I think something like this is pretty likely in worlds where multipolar AGI outcomes happen, and this has an important consequence that I think a lot of futurists miss:
Evolutionary/selective forces become way, way weaker relative to deliberate design/agreements, and more generally the possibility of deal-making is one of the biggest reasons why the future could be very, very good even by your own lights, and it’s one of the most likely ways humans ever have most of what they want according to their values.
This consideration is a part of why I don’t think it matters for us to self-correct, unlike Wei Dai. We already have techniques that are robust under a lot of philosophical uncertainty, we just need to implement them.
A non-trivial amount of futures rely on the assumption of evolutionary/selective forces continuing, and are therefore less probable than they seem based on historical evidence.
Gradual Disempowerment/The Intelligence Curse, at least in it’s more negative/totalizing forms, is a good example of scenarios I expect to be less likely because of this consideration.
Similarly, Burning the Cosmic Commons is another scenario that I give less weight to based on this consideration (Honestly a lot of Robin Hanson-esque futures rely on the assumption of evolution continuing, which I find much less likely than he does)
It is not a coincidence that the futures that are made less likely by deal-making are the futures we’d consider bad, as evolution is misaligned to us, especially at the large scale/long run, and also makes cooperation harder.
I think something like this is pretty likely in worlds where multipolar AGI outcomes happen, and this has an important consequence that I think a lot of futurists miss:
Evolutionary/selective forces become way, way weaker relative to deliberate design/agreements, and more generally the possibility of deal-making is one of the biggest reasons why the future could be very, very good even by your own lights, and it’s one of the most likely ways humans ever have most of what they want according to their values.
This consideration is a part of why I don’t think it matters for us to self-correct, unlike Wei Dai. We already have techniques that are robust under a lot of philosophical uncertainty, we just need to implement them.
A non-trivial amount of futures rely on the assumption of evolutionary/selective forces continuing, and are therefore less probable than they seem based on historical evidence.
Gradual Disempowerment/The Intelligence Curse, at least in it’s more negative/totalizing forms, is a good example of scenarios I expect to be less likely because of this consideration.
Similarly, Burning the Cosmic Commons is another scenario that I give less weight to based on this consideration (Honestly a lot of Robin Hanson-esque futures rely on the assumption of evolution continuing, which I find much less likely than he does)
It is not a coincidence that the futures that are made less likely by deal-making are the futures we’d consider bad, as evolution is misaligned to us, especially at the large scale/long run, and also makes cooperation harder.