Yeah I think there’s some argument to be made that perhaps the ethical thing to do is actually to optimize the preferences which the superintelligence knows the agents would eventually converge on, assuming this is something that can be predicted in advance;
If it would be the case that eventually you know with high certainty that, given ideal reflection processes, people would come to the conclusion that they wish they had more rapidly been converted to some other state, and in fact actually from their reflective perspective they would greatly regret having wasted time slowly coming to that state, or risking some higher variance, less ideal decision process that might be less likely to conclude on what a more ideal process would come to, then maybe the moral thing to do is actually to violate immediate preferences in order to better fulfill well-considered preferences.
I could see a lot of people resisting this on grounds that it violates their rights or something, but I agree there’s probably pretty efficient ways around this with super-persuasion or other similar ASI capabilities if it believed following a fully consensual process was an important constraint…
That said, even if everyone knew this was going to happen and it would be fully consensual, I also agree a lot of people would still be highly panicked by the idea if they hadn’t already been persuaded; I guess in this case he would hope that the ASI would’ve sent out the super-persuasion message in advance to assuage people’s panic… I don’t know haha, I feel like we are expecting a lot from this ASI, but maybe that’s reasonable.
There’s a tricky thing related to child development and non-trans-human economies. A lot of parenting/teaching philosophy is against hovering and helping too much, because there is an expectation of eventually responsible autonomy, that is bootstrapping into place. Failing to build a good tree house the first time, and then doing better on the second one… can help you learn how to build a real house later (after all the older people have died of old age and aren’t around to swoop in and do it for you).
If we have immortal robo angels floating around, that can and will take care of any and all humans literally forever, maybe it is OK for humans to turn into “mere Eloi” who have never had to think or plan or have grit, and so simply lack many of higher capacities of a normal adult human mind in our era. I am proud when I can attend to duties out of my own power, and in accord with my own conscience, and it seems likely that many other humans are too. Also… having a desire to be proud this way might be something that we evolved for, because it pulls us in useful directions.
So IF YOU REJECT THE ELOI OPTION, then… there is still the question of how much to intervene, and how much to not intervene, in exactly which cases… like in terms of “pedagogy balanced against safety”?
For example, if the kid’s tree house is 50 feet up, and will predictably fall down, and the kid will likely die in that fall, then the “valuable end state” of “a smarter kid who has learned how to build a tree house by trial and error” will NOT happen unless someone or something competent swoops in somehow to prevent the predictable tragedy.
Maybe to forbid the construction (if wise enough, but very poor)?
Maybe to install safety stuff fast (if wise and materially rich… but also very busy)?
Maybe to join in the process in a way that sandbags and points at things and gives clues and eventually leads to the child themself making it properly safe the first time (if the parent/angel has wisdom, wealth and free time for such things)?
If one has the resources as a parent for any of these, you could even just explain all three options to the child, and ask the child which kind of help they want… but if the child chooses option #2 over and over every time, that might be heading for a “spoiled child” who will end up as a “mere Eloi”?
In the story, one interpretation is that it is merely and simply murder (and this is being hidden from the children) but another interpretation is that option #2 is being imposed in a way that is a moral error (and maybe “adult” humans of Earth are right to be sad) and another is that option #2 is being imposed due to reasons of budgetary limits, and the humans are too narrow minded to see this.
I assumed that the latter was the case, because like… who are you going to think got things right here? An ASI? Or a bunch of dumb humans (most of whom are barely literate, barely numerate, inalgorate, haven’t studied history or philosophy or economics or higher math, etc, etc, etc).
Interesting!
Yeah I think there’s some argument to be made that perhaps the ethical thing to do is actually to optimize the preferences which the superintelligence knows the agents would eventually converge on, assuming this is something that can be predicted in advance;
If it would be the case that eventually you know with high certainty that, given ideal reflection processes, people would come to the conclusion that they wish they had more rapidly been converted to some other state, and in fact actually from their reflective perspective they would greatly regret having wasted time slowly coming to that state, or risking some higher variance, less ideal decision process that might be less likely to conclude on what a more ideal process would come to, then maybe the moral thing to do is actually to violate immediate preferences in order to better fulfill well-considered preferences.
I could see a lot of people resisting this on grounds that it violates their rights or something, but I agree there’s probably pretty efficient ways around this with super-persuasion or other similar ASI capabilities if it believed following a fully consensual process was an important constraint…
That said, even if everyone knew this was going to happen and it would be fully consensual, I also agree a lot of people would still be highly panicked by the idea if they hadn’t already been persuaded; I guess in this case he would hope that the ASI would’ve sent out the super-persuasion message in advance to assuage people’s panic… I don’t know haha, I feel like we are expecting a lot from this ASI, but maybe that’s reasonable.
There’s a tricky thing related to child development and non-trans-human economies. A lot of parenting/teaching philosophy is against hovering and helping too much, because there is an expectation of eventually responsible autonomy, that is bootstrapping into place. Failing to build a good tree house the first time, and then doing better on the second one… can help you learn how to build a real house later (after all the older people have died of old age and aren’t around to swoop in and do it for you).
If we have immortal robo angels floating around, that can and will take care of any and all humans literally forever, maybe it is OK for humans to turn into “mere Eloi” who have never had to think or plan or have grit, and so simply lack many of higher capacities of a normal adult human mind in our era. I am proud when I can attend to duties out of my own power, and in accord with my own conscience, and it seems likely that many other humans are too. Also… having a desire to be proud this way might be something that we evolved for, because it pulls us in useful directions.
So IF YOU REJECT THE ELOI OPTION, then… there is still the question of how much to intervene, and how much to not intervene, in exactly which cases… like in terms of “pedagogy balanced against safety”?
For example, if the kid’s tree house is 50 feet up, and will predictably fall down, and the kid will likely die in that fall, then the “valuable end state” of “a smarter kid who has learned how to build a tree house by trial and error” will NOT happen unless someone or something competent swoops in somehow to prevent the predictable tragedy.
Maybe to forbid the construction (if wise enough, but very poor)?
Maybe to install safety stuff fast (if wise and materially rich… but also very busy)?
Maybe to join in the process in a way that sandbags and points at things and gives clues and eventually leads to the child themself making it properly safe the first time (if the parent/angel has wisdom, wealth and free time for such things)?
If one has the resources as a parent for any of these, you could even just explain all three options to the child, and ask the child which kind of help they want… but if the child chooses option #2 over and over every time, that might be heading for a “spoiled child” who will end up as a “mere Eloi”?
In the story, one interpretation is that it is merely and simply murder (and this is being hidden from the children) but another interpretation is that option #2 is being imposed in a way that is a moral error (and maybe “adult” humans of Earth are right to be sad) and another is that option #2 is being imposed due to reasons of budgetary limits, and the humans are too narrow minded to see this.
I assumed that the latter was the case, because like… who are you going to think got things right here? An ASI? Or a bunch of dumb humans (most of whom are barely literate, barely numerate, inalgorate, haven’t studied history or philosophy or economics or higher math, etc, etc, etc).