Dylan Evans
Founder and CEO of Projection Point; author, Risk Intelligence
The Great AI Swindle
Smart people often manage to avoid the cognitive errors that bedevil less well-endowed minds. But there are some kinds of foolishness that seem only to afflict the very intelligent. Worrying about the dangers of unfriendly AI is a prime example. A preoccupation with the risks of superintelligent machines is the smart person’s Kool Aid.
This is not to say that superintelligent machines pose no danger to humanity. It is simply that there are many other more pressing and more probable risks facing us this century. People who worry about unfriendly AI tend to argue that the other risks are already the subject of much discussion, and that even if the probability of being wiped out by superintelligent machines is very low, it is surely wise to allocate some brainpower to preventing such an event, given the existential nature of the threat.
Not coincidentally, the problem with this argument was first identified by some of its most vocal proponents. It involves a fallacy that has been termed “Pascal’s mugging,” by analogy with Pascal’s famous wager. A mugger approaches Pascal and proposes a deal: in exchange for the philosopher’s wallet, the mugger will give him back double the amount of money the following day. Pascal demurs. The mugger then offers progressively greater rewards, pointing out that for any low probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet—and a rational person must surely admit there is at least some small chance that such a deal is possible. Finally convinced, Pascal gives the mugger his wallet.
This thought experiment exposes a weakness in classical decision theory. If we simply calculate utilities in the classical manner, it seems there is no way round the problem; a rational Pascal must hand over his wallet. By analogy, even if there is there is only a small chance of unfriendly AI, or a small chance of preventing it, it can be rational to invest at least some resources in tackling this threat.
It is easy to make the sums come out right, especially if you invent billions of imaginary future people (perhaps existing only in software—a minor detail) who live for billions of years, and are capable of far greater levels of happiness than the pathetic flesh and blood humans alive today. When such vast amounts of utility are at stake, who could begrudge spending a few million dollars to safeguard it, even when the chances of success are tiny?
Why do some otherwise very smart people fall for this sleight of hand? I think it is because it panders to their narcissism. To regard oneself as one of a select few far-sighted thinkers who might turn out to be the saviors of mankind must be very rewarding. But the argument also has a very material benefit: it provides some of those who advance it with a lucrative income stream. For in the past few years they have managed to convince some very wealthy benefactors not only that the risk of unfriendly AI is real, but also that they are the people best placed to mitigate it. The result is a clutch of new organizations that divert philanthropy away from more deserving causes. It is worth noting, for example, that Give Well—a non-profit that evaluates the cost-effectiveness of organizations that rely on donations—refuses to endorse any of these self-proclaimed guardians of the galaxy.
But whenever an argument becomes fashionable, it is always worth asking the vital question—Cui bono? Who benefits, materially speaking, from the growing credence in this line of thinking? One need not be particularly skeptical to discern the economic interests at stake. In other words, beware not so much of machines that think, but of their self-appointed masters.
it provides some of those who advance it with a lucrative income stream.
Not me! As I fully expected, I’ve earned less than the minimum wage for my book on the singularity. And I get the impression that most people involved in the singularity movement are earning far less than they could given their skill set.
You say that as if the point of view expressed by Dylan Evans here is one that hasn’t been expressed before. It seems to me more like what until recently was the default reaction to any concerns about unfriendly AI.
I’ve noticed a pattern: Someone implies that some (critical or controversial) position X isn’t represented here, even though X is obviously represented, often by prominent posters in highly up-voted comments.
I think what happens is that some advocates of X literally cannot recognize their own position when it’s presented in a non-tribal manner.
It is worth noting, for example, that Give Well—a non-profit that evaluates the cost-effectiveness of organizations that rely on donations—refuses to endorse any of these self-proclaimed guardians of the galaxy.
GiveWell recommends extremely few charities. Unless you similarly write off the Red Cross, United Way, the Salvation Army, and everyone else GiveWell doesn’t recommend, this looks like motivated skepticism.
It seems to me that there are two key points in Evans’s argument where he makes a controversial claim and needs to justify it, and that at both he kinda cheats.
The first is where he goes from a description of the “Pascal’s Mugging” scenario to saying that that’s a good way to describe concerns over unfriendly AI. (Rather than, e.g., seeing them as analogous to insurance, where one pays a modest but annoying sum for alleged protection against various unlikely but potentially devastating events.) He doesn’t make any attempt at all to justify this; I think he just hopes that the reader won’t notice.
The second is where he suggests that “some of those who advance [UFAI arguments]” are getting a lucrative income stream from doing so. It seems to me that actually awfully few are, and most of those could have got richer faster and more reliably by other more normal means. So if he’s saying about their motives what he seems to be, then again he really owes the reader some justification. Which, again, is not there.
(Maybe there’s a third. I think his last paragraph is just repeating the one that precedes it. But maybe he’s suggesting some other, more powerful “economic interests” at work; if so, it’s not at all clear to me who he has in mind.)
I think the entire core of his argument is a sleight-of-hand between “improbable” and “the kind of absurd improbability involved in Pascal’s wager”, without even (as others have pointed out) giving any arguments for why it’s improbable in the first place.
Why do some otherwise very smart people fall for this sleight of hand? I think it is because it panders to their narcissism. To regard oneself as one of a select few far-sighted thinkers who might turn out to be the saviors of mankind must be very rewarding.
I think this is a bad line of thought even before we get to the hypothesis that people are pushing UFAI risks for the money.
For one thing, people just get things wrong a lot—it doesn’t take bad motivations.
For another, it’s very easy to jump to the conclusion that what seems to be correct to you is so obviously correct that other people must be getting it wrong on purpose.
For a third, even if you’re right that other people are engaged in motivated thinking, you might be wrong about the motivation. For example, concern about UFAI might be driven by anxiety, or by “ooh, shiny! cool idea!” more than by narcissism or money.
advancedatheist, how sure are you of your motivations?
The idea that AI is a low probability risk is one that has some merit, but one doesn’t need a Pascal’s Mugging sort of scenario to consider it to be a problem. If it is only 5 or 10 percent of existential risk in the next century then it is already a serious problem. In general, all existential risks are underfunded by a lot. The only difference with AI is that for a long time it has been even more underfunded than other sources of existential risk.
A booster for getting AI values right is the 2 sidedness of the process. Existential risk and benefit.
To illustrate—You solve poverty, you still have to face climate change, you solve climate change, you still have to face biopathogens, you solve biopathogens, you still have to face nanotech, you solve nanotech, you still have to face SI.
You solve SI correctly, the rest are all done. For people who use the cui bono argument, I think this answer is usually the best one to give.
This assumes that you get a very strong singularity with either a hard take off or a fairly fast takeoff. If someone doesn’t assign that high a probability to AI engaging in recursive self-improvement this argument will be unpersuasive.
As long as the probability of receiving money tomorrow decreases faster than the increase in the amount of promised money, you never have to hand over your wallet.
There’s no reason to think that trend holds, though. If twice the world’s GDP is offered, why would an offer of four times world GDP be less likely to any significant degree, let alone half as likely or less?
Well, someone had to say it:
http://edge.org/response-detail/26073
Dylan Evans Founder and CEO of Projection Point; author, Risk Intelligence
The Great AI Swindle
Smart people often manage to avoid the cognitive errors that bedevil less well-endowed minds. But there are some kinds of foolishness that seem only to afflict the very intelligent. Worrying about the dangers of unfriendly AI is a prime example. A preoccupation with the risks of superintelligent machines is the smart person’s Kool Aid.
This is not to say that superintelligent machines pose no danger to humanity. It is simply that there are many other more pressing and more probable risks facing us this century. People who worry about unfriendly AI tend to argue that the other risks are already the subject of much discussion, and that even if the probability of being wiped out by superintelligent machines is very low, it is surely wise to allocate some brainpower to preventing such an event, given the existential nature of the threat.
Not coincidentally, the problem with this argument was first identified by some of its most vocal proponents. It involves a fallacy that has been termed “Pascal’s mugging,” by analogy with Pascal’s famous wager. A mugger approaches Pascal and proposes a deal: in exchange for the philosopher’s wallet, the mugger will give him back double the amount of money the following day. Pascal demurs. The mugger then offers progressively greater rewards, pointing out that for any low probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet—and a rational person must surely admit there is at least some small chance that such a deal is possible. Finally convinced, Pascal gives the mugger his wallet.
This thought experiment exposes a weakness in classical decision theory. If we simply calculate utilities in the classical manner, it seems there is no way round the problem; a rational Pascal must hand over his wallet. By analogy, even if there is there is only a small chance of unfriendly AI, or a small chance of preventing it, it can be rational to invest at least some resources in tackling this threat.
It is easy to make the sums come out right, especially if you invent billions of imaginary future people (perhaps existing only in software—a minor detail) who live for billions of years, and are capable of far greater levels of happiness than the pathetic flesh and blood humans alive today. When such vast amounts of utility are at stake, who could begrudge spending a few million dollars to safeguard it, even when the chances of success are tiny?
Why do some otherwise very smart people fall for this sleight of hand? I think it is because it panders to their narcissism. To regard oneself as one of a select few far-sighted thinkers who might turn out to be the saviors of mankind must be very rewarding. But the argument also has a very material benefit: it provides some of those who advance it with a lucrative income stream. For in the past few years they have managed to convince some very wealthy benefactors not only that the risk of unfriendly AI is real, but also that they are the people best placed to mitigate it. The result is a clutch of new organizations that divert philanthropy away from more deserving causes. It is worth noting, for example, that Give Well—a non-profit that evaluates the cost-effectiveness of organizations that rely on donations—refuses to endorse any of these self-proclaimed guardians of the galaxy.
But whenever an argument becomes fashionable, it is always worth asking the vital question—Cui bono? Who benefits, materially speaking, from the growing credence in this line of thinking? One need not be particularly skeptical to discern the economic interests at stake. In other words, beware not so much of machines that think, but of their self-appointed masters.
Not me! As I fully expected, I’ve earned less than the minimum wage for my book on the singularity. And I get the impression that most people involved in the singularity movement are earning far less than they could given their skill set.
You say that as if the point of view expressed by Dylan Evans here is one that hasn’t been expressed before. It seems to me more like what until recently was the default reaction to any concerns about unfriendly AI.
I’ve noticed a pattern: Someone implies that some (critical or controversial) position X isn’t represented here, even though X is obviously represented, often by prominent posters in highly up-voted comments.
I think what happens is that some advocates of X literally cannot recognize their own position when it’s presented in a non-tribal manner.
Alternately, claiming novelty is something akin to a bravery debate.
GiveWell recommends extremely few charities. Unless you similarly write off the Red Cross, United Way, the Salvation Army, and everyone else GiveWell doesn’t recommend, this looks like motivated skepticism.
It seems to me that there are two key points in Evans’s argument where he makes a controversial claim and needs to justify it, and that at both he kinda cheats.
The first is where he goes from a description of the “Pascal’s Mugging” scenario to saying that that’s a good way to describe concerns over unfriendly AI. (Rather than, e.g., seeing them as analogous to insurance, where one pays a modest but annoying sum for alleged protection against various unlikely but potentially devastating events.) He doesn’t make any attempt at all to justify this; I think he just hopes that the reader won’t notice.
The second is where he suggests that “some of those who advance [UFAI arguments]” are getting a lucrative income stream from doing so. It seems to me that actually awfully few are, and most of those could have got richer faster and more reliably by other more normal means. So if he’s saying about their motives what he seems to be, then again he really owes the reader some justification. Which, again, is not there.
(Maybe there’s a third. I think his last paragraph is just repeating the one that precedes it. But maybe he’s suggesting some other, more powerful “economic interests” at work; if so, it’s not at all clear to me who he has in mind.)
I think the entire core of his argument is a sleight-of-hand between “improbable” and “the kind of absurd improbability involved in Pascal’s wager”, without even (as others have pointed out) giving any arguments for why it’s improbable in the first place.
I think this is a bad line of thought even before we get to the hypothesis that people are pushing UFAI risks for the money.
For one thing, people just get things wrong a lot—it doesn’t take bad motivations.
For another, it’s very easy to jump to the conclusion that what seems to be correct to you is so obviously correct that other people must be getting it wrong on purpose.
For a third, even if you’re right that other people are engaged in motivated thinking, you might be wrong about the motivation. For example, concern about UFAI might be driven by anxiety, or by “ooh, shiny! cool idea!” more than by narcissism or money.
advancedatheist, how sure are you of your motivations?
The idea that AI is a low probability risk is one that has some merit, but one doesn’t need a Pascal’s Mugging sort of scenario to consider it to be a problem. If it is only 5 or 10 percent of existential risk in the next century then it is already a serious problem. In general, all existential risks are underfunded by a lot. The only difference with AI is that for a long time it has been even more underfunded than other sources of existential risk.
A booster for getting AI values right is the 2 sidedness of the process. Existential risk and benefit.
To illustrate—You solve poverty, you still have to face climate change, you solve climate change, you still have to face biopathogens, you solve biopathogens, you still have to face nanotech, you solve nanotech, you still have to face SI. You solve SI correctly, the rest are all done. For people who use the cui bono argument, I think this answer is usually the best one to give.
This assumes that you get a very strong singularity with either a hard take off or a fairly fast takeoff. If someone doesn’t assign that high a probability to AI engaging in recursive self-improvement this argument will be unpersuasive.
As long as the probability of receiving money tomorrow decreases faster than the increase in the amount of promised money, you never have to hand over your wallet.
There’s no reason to think that trend holds, though. If twice the world’s GDP is offered, why would an offer of four times world GDP be less likely to any significant degree, let alone half as likely or less?