I think it’s worth trying to be much more specific about the mechanism and timeline of this “default outcome”. Does it mean “imminently, standards of living will go down, as we run out of resources and overpopulate and compete”? That seems unlikely to me. Acquiring new resources, or using those resources more efficiently to make our lives better, happens on much shorter time scales than making new humans.
Does it mean “in a few hundred thousand years, space mormons will overpopulate the galaxy and run out of resources, because they had a more effective meme to encourage reproduction”? That is significantly less concerning of a problem to me, and it presupposes that the space mormons can’t intelligently change course in order to protect their standard of living. Which brings us to another question:
Does the “nihilistic optimization” process under scarce resources bottleneck the well-being of general intelligences? Don’t we think that conscious cooperation lets us forecast, understand, and communicate to solve coordination problems? Hasn’t it done so over and over in the past, even when resources have been scarce?
AI x-risk can be expressed as a darwinian competition / gradual disempowerment problem, as can the risks of a permanent authoritarian state (like what chickens live under), but I don’t think it’s the best framing for those issues. e.g. chickens’ standard of living could be significantly improved with the right political coalition, and that coalition won’t be prevented from forming by fundamental darwinian aspects of reality.
I’m purposefully being very vague about the specifics, because I’m fundamentally only trying to make a conceptual point—about how strong the evidence from the outside view should be.
I think it’s worth trying to be much more specific about the mechanism and timeline of this “default outcome”. Does it mean “imminently, standards of living will go down, as we run out of resources and overpopulate and compete”? That seems unlikely to me. Acquiring new resources, or using those resources more efficiently to make our lives better, happens on much shorter time scales than making new humans.
Does it mean “in a few hundred thousand years, space mormons will overpopulate the galaxy and run out of resources, because they had a more effective meme to encourage reproduction”? That is significantly less concerning of a problem to me, and it presupposes that the space mormons can’t intelligently change course in order to protect their standard of living. Which brings us to another question:
Does the “nihilistic optimization” process under scarce resources bottleneck the well-being of general intelligences? Don’t we think that conscious cooperation lets us forecast, understand, and communicate to solve coordination problems? Hasn’t it done so over and over in the past, even when resources have been scarce?
AI x-risk can be expressed as a darwinian competition / gradual disempowerment problem, as can the risks of a permanent authoritarian state (like what chickens live under), but I don’t think it’s the best framing for those issues. e.g. chickens’ standard of living could be significantly improved with the right political coalition, and that coalition won’t be prevented from forming by fundamental darwinian aspects of reality.
I’m purposefully being very vague about the specifics, because I’m fundamentally only trying to make a conceptual point—about how strong the evidence from the outside view should be.