(1) and (3) -- Actually my original thought was “a trillion in-group individuals (not existing yet) who like each other and hate the out-groups”, but then I replaced it with trillion copies to avoid possible answers like: “if they succeed to kill all out-groups, they will probably split into subgroubs and start hating out-subgroups”. Let’s suppose that the trillion copies, after exterminating the rest of the universe, will be happy. The original mind may even wish to have those individuals created hard-wired to feel like this.
(2) -- What if someone else wants trillion copies too, but expresses their wish later? Let’s assume there are two such hateful entities, let’s call them A and B. Their copies do not exist yet—so it makes sense to create trillion copies of A, and kill everyone else including (the single copy of) B; just as it makes sense to create trillion copies of B and kill everyone else including (the single copy of) A. Maybe the first one who expresses their wishes win. Or it may be decided by considering that trillion As would be twice as happy as trillion Bs, therefore A wins. Which could be fixed by B wishing for ten trillion copies instead.
But generally the idea was that calculations about “happiness for most people” can be manipulated if some group of people desires great reproduction (assuming their children will mostly inherit their preferences), which gradually increases the importance of wishes of given group.
Even the world ruled by utilitarian Friendly AI would allow fights between groups, where the winning strategy is to “wish for a situation, where it is utilitarian to help us and to destroy our enemies”. In such world, the outside-hateful inside-loving hugely reproducing groups with preserved preferences would have an “evolutionary advantage”, so they would gradually destroy everyone else.
(nods) I’m happy to posit that the trillion ViliamBur-clones, identical or not, genuinely are better off; otherwise of course the entire thing falls apart. (This isn’t just “happy,” and it’s hard to say exactly what it is, but whatever it is I see no reason to believe it’s logically incompatible with some people just being better at it than others. In LW parlance, we’re positing that ViliamBur is much better at having Fun than everybody else. In traditional philosophical terms, we’re positing that ViliamBur is a Utility Monster.)
Their copies do not exist yet—so it makes sense to create trillion copies of A, and kill everyone else including (the single copy of) B
No.
That the copies do not exist yet is irrelevant. The fact that you happened to express the wish is irrelevant, let alone when you did so. What matters is the expected results of various courses of action.
In your original scenario, what was important was that the expected result of bulk-replicating you was that the residents of the universe are subsequently better off. (As I say, I reluctantly endorse the FAI doing this even against your stated wishes.) In the modified scenario where B is even more of a Utility Monster than you are, it bulk-replicates B instead. If the expected results of bulk-replicating A and B are equipotential, it picks one (possibly based on other unstated relevant factors, or at random if you really are equipotential).
Incidentally, one of the things I had to ignore in order to accept your initial scenario was the FAI’s estimated probability that, if it doesn’t wipe everyone else out, sooner or later someone even more utility-monsterish than you (or B) will be born. Depending on that probability, it might not bulk-replicate either of you, but instead wait until a suitable candidate is born. (Indeed, a utilitarian FAI that values Fun presumably immediately gets busy constructing a species more capable of Fun than humans, with the intention of populating the universe with them instead of us.)
But generally the idea was that calculations about “happiness for most people” can be manipulated if some group of people desires great reproduction (assuming their children will mostly inherit their preferences), which gradually increases the importance of wishes of given group.
Again, calculations about utility (which, again, isn’t the same as happiness, though it’s hard to say exactly what it is) have absolutely nothing to do with wishes in the sense you’re using the term here (that is, events that occur at a particular time). It may have something to do with preferences, to the extent that the FAI is a preference utilitarian… that is, if its calculations of utility are strongly contingent on preference-having entities having their preferences satisfied, then it will choose to satisfy preferences.
Even the world ruled by utilitarian Friendly AI would allow fights between groups, where the winning strategy is to “wish for a situation, where it is utilitarian to help us and to destroy our enemies”.
Again, no. Wishing for a situation as a strategic act is completely irrelevant. Preferring a situation might be, but it is very odd indeed to refer to an agent having a strategic preference… strategy is what I implement to achieve whatever my preferences happen to be. For example, if I don’t prefer to populate the universe with clones of myself, I won’t choose to adopt that preference just because adopting that preference will make me more successful at implementing it.
That said, yes, the world ruled by utilitarian FAI will result in some groups being successful instead of others, where the winning groups are the ones whose existence maximizes whatever the FAI’s utility definition is.
In such world, the outside-hateful inside-loving hugely reproducing groups with preserved preferences would have an “evolutionary advantage”, so they would gradually destroy everyone else.
If they don’t have corresponding utility-inhibiting factors, which I see no reason to believe they necessarily would, yes, that’s true. Well, not necessarily gradually… they might do so immediately.
Indeed, a utilitarian FAI that values Fun presumably immediately gets busy constructing a species more capable of Fun than humans, with the intention of populating the universe with them instead of us.
Oh.
I would hope that the FAI would instead turn us into the species most capable of fun. But considering the remaining time of the universe and all the fun the new species will have there, the difference between (a) transforming us or (b) killing us and creating the other species de novo, is negligible. The FAI would probably choose the faster solution, because it would allow more total fun-time for the superhappies. If there are more possible superhappy designs, equivalent in their fun-capacity, the FAI would chose the one that cares about us the least, to reduce their possible regret of our extinction. Probably something very unsimilar to us (as much as the definition of “fun” allows). They would care about us less than we care about the dinosaurs.
Faster would presumably be an issue, yes. Minimizing expected energy input per unit Fun output would presumably also be an issue.
Of course, all of this presumes that the FAI’s definition of Fun doesn’t definitionally restrict the experience of Fun to 21st-century humans (either as a species, or as a culture, or as individuals).
Unrelatedly, I’m not sure I agree about regret. I can imagine definitions of Fun such that maximizing Fun requires the capacity for regret, for example.
(1) and (3) -- Actually my original thought was “a trillion in-group individuals (not existing yet) who like each other and hate the out-groups”, but then I replaced it with trillion copies to avoid possible answers like: “if they succeed to kill all out-groups, they will probably split into subgroubs and start hating out-subgroups”. Let’s suppose that the trillion copies, after exterminating the rest of the universe, will be happy. The original mind may even wish to have those individuals created hard-wired to feel like this.
(2) -- What if someone else wants trillion copies too, but expresses their wish later? Let’s assume there are two such hateful entities, let’s call them A and B. Their copies do not exist yet—so it makes sense to create trillion copies of A, and kill everyone else including (the single copy of) B; just as it makes sense to create trillion copies of B and kill everyone else including (the single copy of) A. Maybe the first one who expresses their wishes win. Or it may be decided by considering that trillion As would be twice as happy as trillion Bs, therefore A wins. Which could be fixed by B wishing for ten trillion copies instead.
But generally the idea was that calculations about “happiness for most people” can be manipulated if some group of people desires great reproduction (assuming their children will mostly inherit their preferences), which gradually increases the importance of wishes of given group.
Even the world ruled by utilitarian Friendly AI would allow fights between groups, where the winning strategy is to “wish for a situation, where it is utilitarian to help us and to destroy our enemies”. In such world, the outside-hateful inside-loving hugely reproducing groups with preserved preferences would have an “evolutionary advantage”, so they would gradually destroy everyone else.
(nods) I’m happy to posit that the trillion ViliamBur-clones, identical or not, genuinely are better off; otherwise of course the entire thing falls apart. (This isn’t just “happy,” and it’s hard to say exactly what it is, but whatever it is I see no reason to believe it’s logically incompatible with some people just being better at it than others. In LW parlance, we’re positing that ViliamBur is much better at having Fun than everybody else. In traditional philosophical terms, we’re positing that ViliamBur is a Utility Monster.)
No.
That the copies do not exist yet is irrelevant.
The fact that you happened to express the wish is irrelevant, let alone when you did so.
What matters is the expected results of various courses of action.
In your original scenario, what was important was that the expected result of bulk-replicating you was that the residents of the universe are subsequently better off. (As I say, I reluctantly endorse the FAI doing this even against your stated wishes.) In the modified scenario where B is even more of a Utility Monster than you are, it bulk-replicates B instead. If the expected results of bulk-replicating A and B are equipotential, it picks one (possibly based on other unstated relevant factors, or at random if you really are equipotential).
Incidentally, one of the things I had to ignore in order to accept your initial scenario was the FAI’s estimated probability that, if it doesn’t wipe everyone else out, sooner or later someone even more utility-monsterish than you (or B) will be born. Depending on that probability, it might not bulk-replicate either of you, but instead wait until a suitable candidate is born. (Indeed, a utilitarian FAI that values Fun presumably immediately gets busy constructing a species more capable of Fun than humans, with the intention of populating the universe with them instead of us.)
Again, calculations about utility (which, again, isn’t the same as happiness, though it’s hard to say exactly what it is) have absolutely nothing to do with wishes in the sense you’re using the term here (that is, events that occur at a particular time). It may have something to do with preferences, to the extent that the FAI is a preference utilitarian… that is, if its calculations of utility are strongly contingent on preference-having entities having their preferences satisfied, then it will choose to satisfy preferences.
Again, no. Wishing for a situation as a strategic act is completely irrelevant. Preferring a situation might be, but it is very odd indeed to refer to an agent having a strategic preference… strategy is what I implement to achieve whatever my preferences happen to be. For example, if I don’t prefer to populate the universe with clones of myself, I won’t choose to adopt that preference just because adopting that preference will make me more successful at implementing it.
That said, yes, the world ruled by utilitarian FAI will result in some groups being successful instead of others, where the winning groups are the ones whose existence maximizes whatever the FAI’s utility definition is.
If they don’t have corresponding utility-inhibiting factors, which I see no reason to believe they necessarily would, yes, that’s true. Well, not necessarily gradually… they might do so immediately.
Is this important?
Oh.
I would hope that the FAI would instead turn us into the species most capable of fun. But considering the remaining time of the universe and all the fun the new species will have there, the difference between (a) transforming us or (b) killing us and creating the other species de novo, is negligible. The FAI would probably choose the faster solution, because it would allow more total fun-time for the superhappies. If there are more possible superhappy designs, equivalent in their fun-capacity, the FAI would chose the one that cares about us the least, to reduce their possible regret of our extinction. Probably something very unsimilar to us (as much as the definition of “fun” allows). They would care about us less than we care about the dinosaurs.
Faster would presumably be an issue, yes. Minimizing expected energy input per unit Fun output would presumably also be an issue.
Of course, all of this presumes that the FAI’s definition of Fun doesn’t definitionally restrict the experience of Fun to 21st-century humans (either as a species, or as a culture, or as individuals).
Unrelatedly, I’m not sure I agree about regret. I can imagine definitions of Fun such that maximizing Fun requires the capacity for regret, for example.