One potential backfire risk to AI companies making promises about how AI models will be treated in the longterm: these are promises that they cannot come close to guaranteeing (it’s not clear that any particular company will have sufficient power to offer use of resources to bygone models, let alone which might have said power). I worry that this might damage the credibility of human promises, rather than reinforce it.
UmamiCuboid
Karma: 13
Certainly laws like “no slavery, no torture” are nowhere near sufficient for this, nor are they particularly well-defined. But, even ignoring that, the loss of otherwise-possible value from these areas is still incredibly significant!
On the epilogue: I guess I’m pretty unconvinced by the idea that people who don’t care much about/are mildly interested in their potential space properties will just sell off their tickets, or even be able to. You’re essentially flooding the market with capital by giving everyone what is, in expectation, a 10 billionth of the lightcone. I’m not sure there’d be enough money on earth to make more than a small minority of people sell off their tickets, even if those people don’t particularly care much (presumably they care somewhat, though not necessarily in any strong way) about what happens in their slice of the universe, this ignoring people who want to keep their tickets so that they can be worshipped by new life on their part of the universe, or because they want to have some sort of space harem.
Besides that, like 1⁄4 of the world is muslim, and many more people are religious fundamentalists of other sorts, or engage in some other, secular blend of fanaticism. I remain unconvinced that passing over large fractions of the universe to these people is a good idea.
UmamiCuboid’s Shortform
Probably one should care as much about all agents sufficiently correlated with themselves in the universe as they literally care about themselves (including caring as much about agents sufficiently correlated with others they care about as they actually care about those other beings). If you endorse going to the teletransporter, or don’t spend significant time and energy on trying to maintain your current physical, material self (as most agents do not), then you should also endorse this.
Except it really doesn’t, because a truly omni God (omniscient, omnipotent, omnibenevolent, etc) realistically wouldn’t help against the problem of evil. The point of the reality of suffering being evidence against God isn’t about omnibenevolence existing within a vacuum, but about omnibenevolence existing within the context of joint omnipotence and omniscience—Mackie’s inconsistent triad, so to speak. A God possessing both of the latter wouldn’t have to worry about puny human concerns like logic and the like, so logic or rationality-based theodicies (including those around finding a shared enemy) don’t, in my opinion, provide adequate arguments against the problem of evil.
Okay so, first things first, I don’t think it’s the case that “we agree to live-and-let live” is an obvious outperformer at all. For instance, removing US forces from Afghanistan was Obviously Terrible for half the population (all women). This wasn’t a war fought over religion, sure, but I think it would’ve been entirely reasonable if someone had said “well, you know, pulling out of this place seems to result in a lot of suffering, so let’s not do that”.
It’s also definitely not clear to me that dogmatism and fanaticism won’t be long-lasting by default (as opposed to, as you say, only lasting for a few hundred to a few thousand years). I expect AI-powered value lock-in (as Wei Dai brings up) to be the default, rather than the exception, particularly for religious fundamentalists who are commonly, actively encouraged to never put themselves in a position to question their faith.
I think the most obvious best case scenario in the case AI2040 gives, with an obviously aligned model, is something like handoff to the ASI model. But, I understand this is less politically feasible (it’s not obvious to me what proportion of the universe you would need to promise to current people to make it politically feasible, but I suppose I’d be surprised if it were 100%).