Okay so, first things first, I don’t think it’s the case that “we agree to live-and-let live” is an obvious outperformer at all. For instance, removing US forces from Afghanistan was Obviously Terrible for half the population (all women). This wasn’t a war fought over religion, sure, but I think it would’ve been entirely reasonable if someone had said “well, you know, pulling out of this place seems to result in a lot of suffering, so let’s not do that”.
It’s also definitely not clear to me that dogmatism and fanaticism won’t be long-lasting by default (as opposed to, as you say, only lasting for a few hundred to a few thousand years). I expect AI-powered value lock-in (as Wei Dai brings up) to be the default, rather than the exception, particularly for religious fundamentalists who are commonly, actively encouraged to never put themselves in a position to question their faith.
I think the most obvious best case scenario in the case AI2040 gives, with an obviously aligned model, is something like handoff to the ASI model. But, I understand this is less politically feasible (it’s not obvious to me what proportion of the universe you would need to promise to current people to make it politically feasible, but I suppose I’d be surprised if it were 100%).
Okay so, first things first, I don’t think it’s the case that “we agree to live-and-let live” is an obvious outperformer at all. For instance, removing US forces from Afghanistan was Obviously Terrible for half the population (all women). This wasn’t a war fought over religion, sure, but I think it would’ve been entirely reasonable if someone had said “well, you know, pulling out of this place seems to result in a lot of suffering, so let’s not do that”.
It’s also definitely not clear to me that dogmatism and fanaticism won’t be long-lasting by default (as opposed to, as you say, only lasting for a few hundred to a few thousand years). I expect AI-powered value lock-in (as Wei Dai brings up) to be the default, rather than the exception, particularly for religious fundamentalists who are commonly, actively encouraged to never put themselves in a position to question their faith.
I think the most obvious best case scenario in the case AI2040 gives, with an obviously aligned model, is something like handoff to the ASI model. But, I understand this is less politically feasible (it’s not obvious to me what proportion of the universe you would need to promise to current people to make it politically feasible, but I suppose I’d be surprised if it were 100%).