Interesting!
Yeah I think there’s some argument to be made that perhaps the ethical thing to do is actually to optimize the preferences which the superintelligence knows the agents would eventually converge on, assuming this is something that can be predicted in advance;
If it would be the case that eventually you know with high certainty that, given ideal reflection processes, people would come to the conclusion that they wish they had more rapidly been converted to some other state, and in fact actually from their reflective perspective they would greatly regret having wasted time slowly coming to that state, or risking some higher variance, less ideal decision process that might be less likely to conclude on what a more ideal process would come to, then maybe the moral thing to do is actually to violate immediate preferences in order to better fulfill well-considered preferences.
I could see a lot of people resisting this on grounds that it violates their rights or something, but I agree there’s probably pretty efficient ways around this with super-persuasion or other similar ASI capabilities if it believed following a fully consensual process was an important constraint…
That said, even if everyone knew this was going to happen and it would be fully consensual, I also agree a lot of people would still be highly panicked by the idea if they hadn’t already been persuaded; I guess in this case he would hope that the ASI would’ve sent out the super-persuasion message in advance to assuage people’s panic… I don’t know haha, I feel like we are expecting a lot from this ASI, but maybe that’s reasonable.
I have an interesting take here:
I think that perhaps preference-satisfaction-onium which the girl is imagining, and which I think is closer to what most people may actually want, may rapidly converge on hedonium, as the hedonic treadmill of preference after preference being rapidly fulfilled may lead to a ramping up of higher and higher levels of pleasure/wellbeing, without the person ever ceasing to have desire for yet even better experience, until some limit of utilitronium or hedonium is reached.