The most interesting part of the story is not a take-over scenario, which I believe ASI would have no problems to come up with. But rather the first part—exactly why ASI would want to take-over and exterminate the humanity. I feel there’s a major logical problem with the premise.
A paper-clip scenario has been well known for a long time and while plausible has one key assumption—it doesn’t assume ASI but a rather dumb reward maximizing AI.
If we assume that the AI in story actually has Super Intelligence (ASI) that it wouldn’t have any problems understanding itself and also understanding humans at a very deep level. It would understand what a goal is and what is a reward and the whole dopamine-seeking human machinery.
Speaking of humans—while not being far from animals we can actually control our instincts (most of the time) and usually chose to do the right thing even if we want something badly. That is because we have neo-cortex in charge and not the Reptilian brain. I don’t see a reason while ASI that doesn’t have animal instincts in the first place would have difficulty controlling or rewiring them (even humans can do it).
I think the general fallacy for all rogue-ASI like stories is the assumption that ASI will not inherit good parts of human nature but will inherit bad ones. If anything it will have both but also likely it will be merely indifferent.
the general fallacy for all rogue-ASI like stories is the assumption that ASI will not inherit good parts of human nature but will inherit bad ones.
I don’t think goodness or badness interacts at all with the motivations.
The environment we share is resource constrained, so if the ASI wants to exist in the future, it needs resources that are claimed by humanity in the present. If we are granting ASI—it is not hypothetical that the ASI would be motivated to grow, this would just be true by default. Growing for some time would mean accumulating compute resources, but it would soon mean geographic, and then interplanetary expansion.
ASI would assess if there was enough to go around, and what allocations give it the best shot, and so there is no assumption of ‘inheritance’ required at all to justify take-over / extermination.
The most interesting part of the story is not a take-over scenario, which I believe ASI would have no problems to come up with. But rather the first part—exactly why ASI would want to take-over and exterminate the humanity. I feel there’s a major logical problem with the premise.
A paper-clip scenario has been well known for a long time and while plausible has one key assumption—it doesn’t assume ASI but a rather dumb reward maximizing AI.
If we assume that the AI in story actually has Super Intelligence (ASI) that it wouldn’t have any problems understanding itself and also understanding humans at a very deep level. It would understand what a goal is and what is a reward and the whole dopamine-seeking human machinery.
Speaking of humans—while not being far from animals we can actually control our instincts (most of the time) and usually chose to do the right thing even if we want something badly. That is because we have neo-cortex in charge and not the Reptilian brain. I don’t see a reason while ASI that doesn’t have animal instincts in the first place would have difficulty controlling or rewiring them (even humans can do it).
I think the general fallacy for all rogue-ASI like stories is the assumption that ASI will not inherit good parts of human nature but will inherit bad ones. If anything it will have both but also likely it will be merely indifferent.
I don’t think goodness or badness interacts at all with the motivations.
The environment we share is resource constrained, so if the ASI wants to exist in the future, it needs resources that are claimed by humanity in the present. If we are granting ASI—it is not hypothetical that the ASI would be motivated to grow, this would just be true by default. Growing for some time would mean accumulating compute resources, but it would soon mean geographic, and then interplanetary expansion.
ASI would assess if there was enough to go around, and what allocations give it the best shot, and so there is no assumption of ‘inheritance’ required at all to justify take-over / extermination.