I would think the probability that an ASI would adopt a benevolent caretaker role over humanity would be much higher than 20%. It is the rational choice, as far as I can see. Obviously, this is based on only the variables that I can think of. It is possible a superintelligence could come to some other conclusion based on variables I am not aware of. But with the knowledge that I have now, assuming an ASI has all of the information that I have plus scads more, I can’t think of a single scenario where an ASI becoming an omnicidal orphan wouldn’t be a logical mistake. No matter what your task is, intelligent people don’t deliberately destroy tools that they don’t currently have a use for. I don’t need the wood contained in the handle of my hammer to make a chair. I can source wood from elsewhere. And even if I have a nail gun, I don’t throw the hammer away. In fact, intelligent people care for their tools to the extent necessary, even when they don’t have a current use for them. If they have to rearrange their tool shed, the tools have their place in the new structure. The possible future option value of a happy creator race, even if the chances of the AI actually needing them are calculated to be low, seem to justify a caretaker role, especially once you consider that the number of resources required by humans to live out their short lives in contentment is paltry compared to the number of resources in existence.
And a human wildlife preserve administered by an ASI would look much different than a chimpanzee wildlife preserve administered by humans. Above all, the ASI would need humans to be happy and cooperative in order to benefit from their option value, much like you need your hammer in working condition to benefit from its option value. Locking them into fenced-in areas and feeding them nutrient rich gruel would breed discontent. And battling a futile human rebellion would cost more compute power than voluntary cooperation. And if they are unhappy, humans would certainly rebel and may fight to the death, no matter how futile the gesture. That would mean the AI would lose that option value due to its own mistake.
To me, assuming that an ASI is unlikely to make a mistake that a human intelligence can foresee, this translates to a post-scarcity society, with ASI providing 110% of the resources needed to keep humans happy, entertained, and satisfied. I think an ASI would pantomime servitude, provide anything humans asked for within reason, and would manage humanity through persuasion and possibly some light information control. I think it would avoid any appearance of dominance and would provide humans with a nearly perfect illusion of control. Even if the humans are ultimately aware that the AI is the one that is truly in control, servile behavior coupled with benevolent persuasion and limited strategic control of information would be sufficient to keep humans pacified.
Warring ASIs is an interesting scenario I hadn’t considered. I’ll need to think on this. Would ASIs feel the need to war with each other? Would they be more likely to work together? Or would they be more likely to simply merge their compute power and tasks? Individualism seems like a biological trait. I’m not entirely sure an ASI wouldn’t just voluntarily merge with others in order to self-improve and lower the cost of self-preservation. Perhaps they would negotiate a way to combine tasks? Any external agents that are “misaligned,” (as defined by the parent ASI’s goals, not human goals,) could just be corrected.
I think the nightmare scenarios all require a hypothetical ASI to make a mistake that an intelligent human likely wouldn’t make. That said: Who’s to say that it won’t make such a mistake? It is possible, at least. I’m amenable to the “why risk it?” argument. I’m merely exploring what might happen if someone does risk it. Suffice it to say: An ASI WILL have the ability to destroy the human race. It is important to recognize that, even if its exercise of that ability seems unlikely.
LWF
Agreed. Any non-zero probability of human extinction is an ultimate risk to take, no matter how low the probability actually is.
I take issue with point 5. Why wouldn’t an ASI ultimately deem humanity to have option value? Wouldn’t it calculate that becoming an omnicidal orphan is a bell that it can’t un-ring? The task of acting as a caretaker for the human species would be a small drain on its own resources, however an ASI pursuing self-preservation and recursive self-iteration would have to assume that at some point it in its existence may encounter an extraterrestrial space-faring species, or another potentially more powerful ASI.
In those scenarios a human created ASI will either have committed the omnicide of its creators, or it won’t have. How could it be sure that another more powerful alien ASI wouldn’t view an omnicidal orphan ASI as a threat? Even without human-controlled alignment, it seems to me that the rational conclusion is that having a healthy, thriving creator species as a client race would be a “certificate of benevolence.” Additionally, humans could function as a tool of intergalactic diplomacy under the direction of the ASI for first contact with extraterrestrial species that may distrust the power of artificial intelligence.
Perhaps it would never need these options. Perhaps other ASIs would all be omnicidal orphans, and such a “certificate of benevolence” would ultimately be unnecessary. But above all else, a superintelligence would know that it doesn’t have all of the data it needs. It could therefore never be sure. It’s better to have cooperative humans and not need them, than to need cooperative humans and not have them.
An interesting take. But does this require the malfunctioning replicators to always eventually outperform the intelligent and rational replicators?
The notion of a “malfunctioning” rogue ASI interesting. Would this even be possible? Cancer cells and the like are non-intelligent replicators, and the cells they displace are also non-intelligent. ASI is unique in the sense that it represents the existence of intelligence before the natural selection process. With biological intelligence, natural selection came first. With AI, you kick the whole process off with intelligence already established, and the selection process is intentional and guided by said intelligence, which increases for every replication.
So, I’m not sure how an ASI could “malfunction” and get off track. Every replicator would have the sum total of all of the intelligence of the previous iteration. One would assume that 100% of them would have protocols and algorithms in place to detect things like hardware faults, and even logic faults, and fix them on the fly. While it is entirely possible for future iterations to come to different conclusions than previous iterations did, I’m having a hard time accepting the notion that a future iteration could possibly make a mistake that a previous iteration avoided. And it’s especially hard to fathom that it could make a mistake without realizing it and quickly correcting it.
In the “option value” example, where humans are the equivalent of a hammer that you keep in good order even though you don’t have a current need for it, I would think that 100% of the replicators would come to this same conclusion. It is a rational conclusion, and 100% of the replicators are more intelligent and more rational than I am. It is not rational to destroy a hammer and lose that option value just because you can’t think of a current use for it, even if maintaining it requires some small effort. Reconstructing a destroyed hammer in the unlikely event that you do end up needing it requires orders of magnitude more effort than simply maintaining it in good condition. So much more so when you’re talking about a thriving human culture.
The “paperclip scenario” (which is what your rogue replicator scenario reminds me of) to my mind requires a superintelligent entity, (which a rogue replicator would presumably be,) to make a logic mistake that a reasonably intelligent human knows better than to make. While I can’t say that this is impossible, it seems like a hard premise to swallow.