Thank you for posting this, as I find it helpful for practicing my own skills of argumentation. Here are my brief counterarguments to your counterarguments, I’d appreciate it if anyone could point out any flaws in my logic:
A. Contra “superhuman AI systems will be goal-directed” As far as I understand it, “intelligence” is the ability to achieve one’s goals through reasoning and making plans, so a highly intelligent system is goal-directed by definition. Less goal-directed AIs are certainly possible, but they must necessarily be considered less intelligent—the thermometer example illustrates this. Therefore, a less goal-directed AI will always lose in competition against a more goal-directed one.
B. Contra “goal-directed AI systems’ goals will be bad” The supposed counterexample of artificially generated human faces is in fact a case in point in my opinion. These faces aren’t like humans at all. They’re not three-dimensional. They’re not moving. They don’t talk. They don’t smell. They’re not soft and don’t radiate warmth. Oh, we didn’t mention that was important, right? We just gave the AI a reward function that enabled it to learn how to generate pictures that look like photographs of real people. If that’s what we want, then little differences on the pixel level probably don’t matter much. The differences between the paperclips Bostrom’s paperclip maximizer makes and a perfect paperclip probably won’t matter much, either. To put it another way, these fake humans are only “good” if we lower our expectations to the point where they’re already met.
C. Contra “superhuman AI would be sufficiently superior to humans to overpower humanity” Even if “human success isn’t from individual intelligence”, this doesn’t mean that human intelligence is not the decisive factor making us the dominant species. Individual intelligence is what enables collective intelligence in the first place. I agree that humans shouldn’t be seen as a universal benchmark for intelligence, but that only means that the bar for developing an uncontrollable AI may be even lower. It took us humans more than 2,000 years to collectively master Go. It took AlphaGo Zero three days from scratch to beat us. AI may one day be sufficiently good at manipulating and controlling humans to take over the world even without being “superintelligent” in all aspects. It could be way more intelligent in the relevant ways, like AlphaGo Zero compared to a child learning to play Go. I believe there is no upper boundary for manipulation skills and other forms of gaining power. So whether intelligence is an overwhelming advantage is probably a matter of scale.
However AI systems have one serious disadvantage as employees of humans: they are intrinsically untrustworthy, while we don’t understand them well enough to be clear on what their values are or how they will behave in any given case. Even if they did perform as well as humans at some task, if humans can’t be certain of that, then there is reason to disprefer using them.
Really? Look at how we use AI today, e.g. in letting it decide what we see, hear and believe, who gets on parole from prison, and who gets a loan. It seems to me that humans tend to trust AI already more than other humans, in particular if they don’t understand how it works.
I have some goals. For instance, I want some good romance. My guess is that trying to take over the universe isn’t the best way to achieve this goal. The same goes for a lot of my goals, it seems to me. Possibly I’m in error, but I spend a lot of time pursuing goals, and very little of it trying to take over the universe.
Imagine you had a magic wand or a genie in a bottle that would fulfill every wish you could dream of. Would you use it? If so, you’re incentivized to take over the world, because the only possible way of making every wish come true is absolute power over the universe. The fact that you normally don’t try to achieve that may have to do with the realization that you have no chance. If you had, I bet you’d try it. I certainly would, if only so I could stop Putin. But would me being all-powerful be a good thing for the rest of the world? I doubt it.
D. Contra the whole argument No, AI is not like a corporation run by humans. AI is more like an alien life form. It does not have intrinsic human motives and values. We may be able to tame it or to give it a beneficial goal, but unless we do, if it can, it will transform the world in very weird and probably unforeseen ways. Apart from that, corporations are currently wreaking a lot of havoc on the world (e.g. climate change), which is a good example of how difficult it is to give a powerful entity a beneficial goal.
As far as I understand it, “intelligence” is the ability to achieve one’s goals through reasoning and making plans, so a highly intelligent system is goal-directed by definition. Less goal-directed AIs are certainly possible, but they must necessarily be considered less intelligent—the thermometer example illustrates this. Therefore, a less goal-directed AI will always lose in competition against a more goal-directed one.
Your argument seems to be:
Definitionally, intelligence is the ability to achieve one’s goals.
Less goal-directed systems are less intelligent.
Less intelligent systems will always lose in competition.
Less goal directed systems will always lose in competition.
Defining intelligence as goal-directedness doesn’t do anything for your argument. It just kicks the can down the road. Why will less intelligent (under your definition, goal directed) always lose in competition?
Imagine you had a magic wand or a genie in a bottle that would fulfill every wish you could dream of. Would you use it? If so, you’re incentivized to take over the world, because the only possible way of making every wish come true is absolute power over the universe. The fact that you normally don’t try to achieve that may have to do with the realization that you have no chance. If you had, I bet you’d try it. I certainly would, if only so I could stop Putin. But would me being all-powerful be a good thing for the rest of the world? I doubt it.
Romance is a canonical example of where you really don’t want to be all powerful (if real romance is what you want). Romance could not exist if your romantic partner always predictably did everything you ever wanted. The whole point is they are a different person, with different wishes, and you have to figure out how to navigate that and its unpredictabilities. That is the “fun” of romance. So no, I don’t think everyone would really use that magic wand.
Defining intelligence as goal-directedness doesn’t do anything for your argument. It just kicks the can down the road. Why will less intelligent (under your definition, goal directed) always lose in competition?
Admittedly, my reply to A was a bit short. I only wanted to point out that intelligence is closely linked to goal-directedness, not that they’re the same thing (heat-seeking missiles are stupid, but very goal-directed entities, for example). A very intelligent system without a goal would just sit around, doing nothing. It might be able to potentially act intelligently, but without a goal it would behave like an unintelligent system. “Always” may be too strong a word, but if system X is more intelligent and wants to reach a conflicting goal much more than system Y, chances are that system X will get what it wants.
Romance is a canonical example of where you really don’t want to be all powerful (if real romance is what you want). Romance could not exist if your romantic partner always predictably did everything you ever wanted.
I disagree. Being all-powerful does not mean always doing everything you want, or everything your partner wants. It means being able to do whatever you want, or maybe more importantly, whatever you feel you need to do. If, for example, I needed the magic wand to prevent the untimely death of someone I love, I would use it without a second thought.
The whole point is they are a different person, with different wishes, and you have to figure out how to navigate that and its unpredictabilities. That is the “fun” of romance.
I tend to agree, but I guess there are many people who have been less lucky in their relationships than I have, being happily together with my wife for more than 44 years. :)
So no, I don’t think everyone would really use that magic wand.
Maybe not everyone and certainly not all the time, but I’m quite sure that most people would use it at least once in a while.
Thank you for posting this, as I find it helpful for practicing my own skills of argumentation. Here are my brief counterarguments to your counterarguments, I’d appreciate it if anyone could point out any flaws in my logic:
A. Contra “superhuman AI systems will be goal-directed”
As far as I understand it, “intelligence” is the ability to achieve one’s goals through reasoning and making plans, so a highly intelligent system is goal-directed by definition. Less goal-directed AIs are certainly possible, but they must necessarily be considered less intelligent—the thermometer example illustrates this. Therefore, a less goal-directed AI will always lose in competition against a more goal-directed one.
B. Contra “goal-directed AI systems’ goals will be bad”
The supposed counterexample of artificially generated human faces is in fact a case in point in my opinion. These faces aren’t like humans at all. They’re not three-dimensional. They’re not moving. They don’t talk. They don’t smell. They’re not soft and don’t radiate warmth. Oh, we didn’t mention that was important, right? We just gave the AI a reward function that enabled it to learn how to generate pictures that look like photographs of real people. If that’s what we want, then little differences on the pixel level probably don’t matter much. The differences between the paperclips Bostrom’s paperclip maximizer makes and a perfect paperclip probably won’t matter much, either. To put it another way, these fake humans are only “good” if we lower our expectations to the point where they’re already met.
C. Contra “superhuman AI would be sufficiently superior to humans to overpower humanity”
Even if “human success isn’t from individual intelligence”, this doesn’t mean that human intelligence is not the decisive factor making us the dominant species. Individual intelligence is what enables collective intelligence in the first place. I agree that humans shouldn’t be seen as a universal benchmark for intelligence, but that only means that the bar for developing an uncontrollable AI may be even lower. It took us humans more than 2,000 years to collectively master Go. It took AlphaGo Zero three days from scratch to beat us. AI may one day be sufficiently good at manipulating and controlling humans to take over the world even without being “superintelligent” in all aspects. It could be way more intelligent in the relevant ways, like AlphaGo Zero compared to a child learning to play Go. I believe there is no upper boundary for manipulation skills and other forms of gaining power. So whether intelligence is an overwhelming advantage is probably a matter of scale.
Really? Look at how we use AI today, e.g. in letting it decide what we see, hear and believe, who gets on parole from prison, and who gets a loan. It seems to me that humans tend to trust AI already more than other humans, in particular if they don’t understand how it works.
Imagine you had a magic wand or a genie in a bottle that would fulfill every wish you could dream of. Would you use it? If so, you’re incentivized to take over the world, because the only possible way of making every wish come true is absolute power over the universe. The fact that you normally don’t try to achieve that may have to do with the realization that you have no chance. If you had, I bet you’d try it. I certainly would, if only so I could stop Putin. But would me being all-powerful be a good thing for the rest of the world? I doubt it.
D. Contra the whole argument
No, AI is not like a corporation run by humans. AI is more like an alien life form. It does not have intrinsic human motives and values. We may be able to tame it or to give it a beneficial goal, but unless we do, if it can, it will transform the world in very weird and probably unforeseen ways. Apart from that, corporations are currently wreaking a lot of havoc on the world (e.g. climate change), which is a good example of how difficult it is to give a powerful entity a beneficial goal.
Your argument seems to be:
Definitionally, intelligence is the ability to achieve one’s goals.
Less goal-directed systems are less intelligent.
Less intelligent systems will always lose in competition.
Less goal directed systems will always lose in competition.
Defining intelligence as goal-directedness doesn’t do anything for your argument. It just kicks the can down the road. Why will less intelligent (under your definition, goal directed) always lose in competition?
Romance is a canonical example of where you really don’t want to be all powerful (if real romance is what you want). Romance could not exist if your romantic partner always predictably did everything you ever wanted. The whole point is they are a different person, with different wishes, and you have to figure out how to navigate that and its unpredictabilities. That is the “fun” of romance. So no, I don’t think everyone would really use that magic wand.
Thank you very much for your input!
Admittedly, my reply to A was a bit short. I only wanted to point out that intelligence is closely linked to goal-directedness, not that they’re the same thing (heat-seeking missiles are stupid, but very goal-directed entities, for example). A very intelligent system without a goal would just sit around, doing nothing. It might be able to potentially act intelligently, but without a goal it would behave like an unintelligent system. “Always” may be too strong a word, but if system X is more intelligent and wants to reach a conflicting goal much more than system Y, chances are that system X will get what it wants.
I disagree. Being all-powerful does not mean always doing everything you want, or everything your partner wants. It means being able to do whatever you want, or maybe more importantly, whatever you feel you need to do. If, for example, I needed the magic wand to prevent the untimely death of someone I love, I would use it without a second thought.
I tend to agree, but I guess there are many people who have been less lucky in their relationships than I have, being happily together with my wife for more than 44 years. :)
Maybe not everyone and certainly not all the time, but I’m quite sure that most people would use it at least once in a while.