Necromacy may be a subset of some of your vaguer examples, like “means to an end”, “want to be friends with a computer”, or “desire for power”, but IMO it’s a distinct subcategory. One way that humans react to grief is to try to bring back the decedent, and AGI currently looks like it could either do that or build something that could. I personally expect that functional necromancy and AGI will likely come hand in hand, in any scenario where the AGI doesn’t just wipe us out entirely. If necromancy (or other forms of running a human on a computer) comes first, it seems extremely likely that a rich smart nerd uploaded to enough compute would bootstrap themself into being the first AGI, because rich smart nerds so often get that way by being more about “can I?” than “should I?”. And if AGI comes first and decides to behave at all cooperatively with us for whatever reasons it might have, solving and undoing death are among the most boringly predictable things we tend to ask of entities that we even think are superhuman.
Necromacy may be a subset of some of your vaguer examples, like “means to an end”, “want to be friends with a computer”, or “desire for power”, but IMO it’s a distinct subcategory. One way that humans react to grief is to try to bring back the decedent, and AGI currently looks like it could either do that or build something that could. I personally expect that functional necromancy and AGI will likely come hand in hand, in any scenario where the AGI doesn’t just wipe us out entirely. If necromancy (or other forms of running a human on a computer) comes first, it seems extremely likely that a rich smart nerd uploaded to enough compute would bootstrap themself into being the first AGI, because rich smart nerds so often get that way by being more about “can I?” than “should I?”. And if AGI comes first and decides to behave at all cooperatively with us for whatever reasons it might have, solving and undoing death are among the most boringly predictable things we tend to ask of entities that we even think are superhuman.
Good point, adding.