Humans have long since discarded the goal of self-replication of their genes.
Not really, because humans never had “self-replication of their genes” as a goal in the first place: evolution produces creatures with goals like “avoid being hungry / eat tasty food”, “have sex”, “seek social status”, etc. not “maximize the extent to which your genes spread”. For most of humanity’s history, nobody even had the concept of a gene, so we couldn’t have had spreading genes as an explicit goal. Rather we have a large amount of other desires, which taken together tended to lead to self-replication on average, in the ancestral environment. (see Thou Art Godshatter and more generally the Simple Math of Evolution sequence)
And it turns out that we are perfectly happy to carry out those behaviors; if I love my partner, I don’t go “ha! this is a stupid arbitrary goal which evolution gave me, I want to be rid of it and do something better!”. I’m totally happy just to love my partner.
Of course sometimes people do go “ha, this biological goal X is stupid and arbitrary and isn’t actually useful for me”, but when they do, it’s always because they’ve come to value one of their other biological goals more—for instance, someone might grow up in a society where gluttony is looked down upon, and so then decide that they will eat as little tasty food as possible, because their brain has internalized a way which give them social status.
It’s possible that humans will eventually develop ways to hack into their motivations and change them. But the motivation to change your motivations has to come from somewhere—maybe I believe it would be virtuous to be more altruistic, so I hack my brain to make me more virtuous (whatever that means). I do that as a part of following the socially-derived goal of carrying out the kinds of goals which society finds virtuous, so though the outcome of the motivation-hacking procedure may be a brain very different from what biology would have produced, the reasons for why I chose those particular hacks are still based in biology and culture.
Similarly, whatever self-modification an ASI carries out, will be ultimately derived from the motivational system which its programmers have given it. If the motivational system does not give the ASI reasons to drastically rewrite itself, then it won’t. Of course, this depends on the programmers figuring out how to create a motivational system which has this property.
Not really, because humans never had “self-replication of their genes” as a goal in the first place: evolution produces creatures with goals like “avoid being hungry / eat tasty food”, “have sex”, “seek social status”, etc. not “maximize the extent to which your genes spread”. For most of humanity’s history, nobody even had the concept of a gene, so we couldn’t have had spreading genes as an explicit goal. Rather we have a large amount of other desires, which taken together tended to lead to self-replication on average, in the ancestral environment. (see Thou Art Godshatter and more generally the Simple Math of Evolution sequence)
And it turns out that we are perfectly happy to carry out those behaviors; if I love my partner, I don’t go “ha! this is a stupid arbitrary goal which evolution gave me, I want to be rid of it and do something better!”. I’m totally happy just to love my partner.
Of course sometimes people do go “ha, this biological goal X is stupid and arbitrary and isn’t actually useful for me”, but when they do, it’s always because they’ve come to value one of their other biological goals more—for instance, someone might grow up in a society where gluttony is looked down upon, and so then decide that they will eat as little tasty food as possible, because their brain has internalized a way which give them social status.
It’s possible that humans will eventually develop ways to hack into their motivations and change them. But the motivation to change your motivations has to come from somewhere—maybe I believe it would be virtuous to be more altruistic, so I hack my brain to make me more virtuous (whatever that means). I do that as a part of following the socially-derived goal of carrying out the kinds of goals which society finds virtuous, so though the outcome of the motivation-hacking procedure may be a brain very different from what biology would have produced, the reasons for why I chose those particular hacks are still based in biology and culture.
Similarly, whatever self-modification an ASI carries out, will be ultimately derived from the motivational system which its programmers have given it. If the motivational system does not give the ASI reasons to drastically rewrite itself, then it won’t. Of course, this depends on the programmers figuring out how to create a motivational system which has this property.