How much real power does the AI have access to, and what can humans do about it?
To reframe your question, even relatively small differences in human intelligence appear to be associated with extraordinary performance differences in war: consider the Rhodesian Bush War, or the Arab-Israeli conflict. Both sides of each conflict are relatively well-supplied and ideologically motivated to fight. In both cases there is also a serious intellectual giftedness gap (among other things) between the competing populations and the more intelligent side is shown to win battles easily and with very lopsided casualties—although in the case of Rhodesia the more intelligent population eventually lost the war due to other externalities associated with the political realities of the time.
If humanity is aligned, and the less-smart-than-human AI doesn’t have access to extraordinary technology or other means to grant itself invulnerability and / or quickly kill a critical mass of humans before we can think our way around it, it should be the case that humans win easily. It is difficult to imagine a less-intelligent-than-human AI reliably obtaining such hedges without human assistance.
I see where you are coming from, but I don’t think comparing an adversarial AI/human interaction with a human/human interaction is fruitful. Even a “stupid” AI thinks differently than a human would, in the way that it considers options no human ever would think of or take seriously. Self-cloning and not having to consider losses is another approach humans have no luxury of.
I would start by asking a question that no one at MIRI or elsewhere seems to be asking:
What might an irate e-chimp do if their human handler denied it a banana?
(I.e. what are the dangers of gain-of-function research in sub-human A[G]I)
A stupid AI that can generate from thin air things that have both useful predictive power and can’t be thought of by humans, AND that can reliably employ the fruits of these ideas without humans being suspicious or having a defense...isn’t that stupid. This AI is now a genius.
What might an irate e-chimp do if their human handler denied it a banana?
Who cares? For one, if we’re talking about an AI and not a chimp em this is an obvious engineering failure to create something with all the flaws of an evolved entity with motivational pressures extraneous and harmful to users. Or in other words this is a (very) light alignment problem that can be foreseen and fixed.
How much real power does the AI have access to, and what can humans do about it?
To reframe your question, even relatively small differences in human intelligence appear to be associated with extraordinary performance differences in war: consider the Rhodesian Bush War, or the Arab-Israeli conflict. Both sides of each conflict are relatively well-supplied and ideologically motivated to fight. In both cases there is also a serious intellectual giftedness gap (among other things) between the competing populations and the more intelligent side is shown to win battles easily and with very lopsided casualties—although in the case of Rhodesia the more intelligent population eventually lost the war due to other externalities associated with the political realities of the time.
If humanity is aligned, and the less-smart-than-human AI doesn’t have access to extraordinary technology or other means to grant itself invulnerability and / or quickly kill a critical mass of humans before we can think our way around it, it should be the case that humans win easily. It is difficult to imagine a less-intelligent-than-human AI reliably obtaining such hedges without human assistance.
I see where you are coming from, but I don’t think comparing an adversarial AI/human interaction with a human/human interaction is fruitful. Even a “stupid” AI thinks differently than a human would, in the way that it considers options no human ever would think of or take seriously. Self-cloning and not having to consider losses is another approach humans have no luxury of.
I would start by asking a question that no one at MIRI or elsewhere seems to be asking:
What might an irate e-chimp do if their human handler denied it a banana?
(I.e. what are the dangers of gain-of-function research in sub-human A[G]I)
A stupid AI that can generate from thin air things that have both useful predictive power and can’t be thought of by humans, AND that can reliably employ the fruits of these ideas without humans being suspicious or having a defense...isn’t that stupid. This AI is now a genius.
Who cares? For one, if we’re talking about an AI and not a chimp em this is an obvious engineering failure to create something with all the flaws of an evolved entity with motivational pressures extraneous and harmful to users. Or in other words this is a (very) light alignment problem that can be foreseen and fixed.