Crocker’s rules declared, because I expect this may agitate some people:
(1) the claim that intelligence explosion is plausible, (2) the claim that intelligence explosion is likely to occur within the next 150 years, and (3) the claim that intelligence explosion would have a massive impact on civilization.
I accept (1) and (3). Where I depart somewhat from the LW consensus is in the belief that anyone is going to accept the idea that the singularity (in its intelligence explosion form) should go ahead, without some important intervening stages that are likely to last for longer than 150 years.
CEV is a bad idea. I am sympathetic towards the mindset of the people who advocate it, but even I would be in the pitchfork-wielding gang if it looked like someone was actually going to implement it. Try to imagine that this was actually going to happen next year, rather than being a fun thing discussed on an internet forum – beware far mode bias.
To quote Robin Hanson in a recent OB post:
Immersing ourselves in images of things large in space, time, and social distance puts us into a “transcendant” far mode where positive feelings are strong, our basic ideals are more visible than practical constraints, and where analysis takes a back seat to metaphor.
I don’t trust fallible human programmers to implement soundly “knowing more”, “thinking faster” and “growing up together”, and deal with the problems of “muddle”, “spread” and “distance”. The idea of a “last judge” as a safety measure seems like a sticking plaster on a gaping wound. Neither do I accept that including all of humanity is anything other than misplaced idealism. Some people seem to think that even a faulty CEV initial dynamic magically corrects itself into a good one; that might happen, but not with nearly a high enough probability.
Another problem that has been scarcely discussed: what happens if, as Eliezer’s CEV document suggests might happen, the thing shuts itself down or the last judge decides it isn’t safe? And the second time we try it, too?
But the problem remains that a superintelligence needs a full set of human values in order for it to be safe. I don’t see any other tenable proposals for implementing this apart from CEV, therefore I conclude that building a recursively improving superintelligence is basically just unsafe, given present human competence levels. Given that fact, to conclude that because we are likely to obtain the means to bring about a (positive or negative) singularity at some point we cannot prevent it from happening indefinitely is like saying that because we possess nuclear technology we can’t prevent a nuclear extinction event from happening indefinitely. If FAI is an “impossible” challenge and NAI (No AI) is merely very difficult, there is something to recommend NAI.
That doesn’t mean to say that I disprove of what Eliezer et al are doing. The singularity is definitely an extremely important thing to be discussing. I just think that the end product is likely to be widespread recognition of the peril of playing around with AI, and this (along with appropriately severe action taken to reduce the peril) is just as much a solution to Yudkowsky’s fear that a bunch of above-average AI scientists can “learn from each other’s partial successes and accumulate hacks as a community” as is trying to beat them to the punch by rushing to create a positive singularity.
Although this is unfair there is probably some truth to the idea that people who devote their lives to studying AI and the intelligence explosion are likely to be biased towards solutions to the problem in which their work achieves something really positive, rather than merely acting as a warning. That is not to pre-judge the issue, but merely to recommend that a little more skepticism than normal is due.
On the other hand there is another tenable approach to the singularity that is less widely recognised here. Wei Dai’s posts here and here seem very sensible to me; he suggests that intelligence enhancement should have priority over FAI research:
Given that there are known ways to significantly increase the number of geniuses (i.e., von Neumann level, or IQ 180 and greater), by cloning or embryo selection, an obvious alternative Singularity strategy is to invest directly or indirectly in these technologies, and to try to mitigate existential risks (for example by attempting to delay all significant AI efforts) until they mature and bear fruit (in the form of adult genius-level FAI researchers). [...]
The chances of success seem higher, and if disaster does occur as a result of the intelligence amplification effort, we’re more likely to be left with a future that is at least partly influenced by human values.
He quotes Eliezer as having said this (from pages 31-35 here):
If AI is naturally far more difficult than intelligence enhancement, no harm done; if building a 747 is naturally easier than inflating a bird, then the wait could be fatal. There is a relatively small region of possibility within which deliberately not working on Friendly AI could possibly help, and a large region within which it would be either irrelevant or harmful.
Wei Dai points out that it is worth distinguishing the ease of creating uFAI in comparison to FAI, rather than lumping these together as “AI”.
I also think that the difference in outcomes between “deliberately not working on Friendly AI” and “treating unsupervised AI work as a terrible crime” are worth distinguishing.
Even if human intelligence enhancement is possible, there are real, difficult safety considerations; I would have to seriously ask whether we wanted Friendly AI to precede intelligence enhancement, rather than vice versa.
This depends on the probability one assigns to CEV working. My probability that it would work given present human competence levels is low, and my probability that anyone would actually let it happen is very low.
The benefit of intelligence enhancement is that changes can be as unhurried and incremental as one likes (assuming that the risk of someone building uFAI is not considered to be imminent, due to stringent security measures); CEV is more a leap of faith.
Seconded—I’d like to see some material from lukeprog or somebody else at SI addressing these kinds of concerns. A “Criticisms of CEV” page maybe?
[Edit: just to clarify, I wasn’t seconding the part about the pitchforks and I’m not sure that either IA or an AGI ban is an obviously better strategy. But I agree with everything else here]
Given that fact, to conclude that because we are likely to obtain the means to bring about a (positive or negative) singularity at some point we cannot prevent it from happening indefinitely is like saying that because we possess nuclear technology we can’t prevent a nuclear extinction event from happening indefinitely.
The problem is that for a nuclear explosion to take place, the higher ups of some country have to approve it.
For AGI, all that has to occur is code is leaked or hacked, then preventing the AGI from being implemented somewhere is an impossible task. And right now, no major online institution in the entire world is safe from being hacked.
Perhaps that is a motivation to completely (and effectively) prohibit research in the direction of creating superintelligent AI, licensed or otherwise, and concentrate entirely on human intelligence enhancement.
Ignoring how difficult this would be (due to ease of secret development considering it would largely consist of code rather than easy to track hardware) even if every country in the world WANTED to cooperate on it, the real problem comes from the high potential value of defecting. Much like nuclear weapons development, it would take a lot more than sanctions to convince a rogue nation NOT to try and develop AI for its own benefit, should this become a plausible course of action.
the real problem comes from the high potential value of defecting
What would anyone think they stood to gain from creating an AI, if they understood the consequences as described by Yudkowsky et al?
The situation is not “much like nuclear weapons development”, because nuclear weapons are actually a practical warfare device, and the comparison was not intended to imply this similarity. I just meant to say that we manage to keep nukes out of the hands of terrorists, so there is reason to be optimistic about our chances of preventing irresponsible or crazy people from successfully developing a recursively self-improving AI—it is difficult, but if creating and successfully implementing a provably safe FAI (without prior intelligence enhancement) is hopelessly difficult—even if only because the large majority of people wouldn’t consent to it—then it may still be our best option.
The same things WE hope to gain from creating AI. I do not trust north korea (for example) to properly decide on the relative risks/rewards of any given course of action it can undertake.
OK but it isn’t hard (or wouldn’t be in the context we are discussing) to come to the understanding that creating an intelligence explosion renders the interests of any particular state irrelevant. I’ve seen no evidence that the North Koreans are that crazy.
The problem would be people who think that something like CEV, implemented by present-day humans, is actually safe—and the people liable to believe that are more likely to be the type of people found here, not North Koreans or other non-Westerners.
I’d also be interested in hearing your opinion on the security concerns should we attempt to implement CEV, and find that it shut itself down or produced an unacceptable output.
OK but it isn’t hard (or wouldn’t be in the context we are discussing) to come to the understanding that creating an intelligence explosion renders the interests of any particular state irrelevant.
If you’re correct, then the best way to stave off the optimists from trying is to make an indisputable case for pessimism and disseminate it widely. Otherwise, eventually someone else will get optimistic, and won’t see why they shouldn’t give it a go.
I expect that once recognition of the intelligence explosion as a plausible scenario becomes mainstream, pessimism about the prospects of (unmodified) human programmers safely and successfully implementing CEV or some such thing will be the default, regardless of what certain AI researchers claim.
In that case, optimists are likely to have their activities forcefully curtailed. If this did not turn out to be the case, then I would consider “pro-pessimism” activism to change that state of affairs (assuming nothing happens to change my mind between now and then). At the moment however I support the activities of the Singularity Institute, because they are raising awareness of the problem (which is a prerequisite for state involvement) and they are highly responsible people. The worst state of affairs would be one in which no-one recognised the prospect of an intelligence explosion until it was too late.
ETA: I would be somewhat more supportive of a CEV in which only a select (and widely admired and recognised) group of humans was included. This seems to create an opportunity for the CEV initial dynamic implementation to be a compromise between intelligence enhancement and ordinary CEV, i.e. a small group of humans can be “prepared” and studied very carefully before the initial dynamic is switched on.
So really it’s a complex situation, and my post above probably failed to express the degree of ambivalence that I feel regarding this subject.
Crocker’s rules declared, because I expect this may agitate some people:
I accept (1) and (3). Where I depart somewhat from the LW consensus is in the belief that anyone is going to accept the idea that the singularity (in its intelligence explosion form) should go ahead, without some important intervening stages that are likely to last for longer than 150 years.
CEV is a bad idea. I am sympathetic towards the mindset of the people who advocate it, but even I would be in the pitchfork-wielding gang if it looked like someone was actually going to implement it. Try to imagine that this was actually going to happen next year, rather than being a fun thing discussed on an internet forum – beware far mode bias. To quote Robin Hanson in a recent OB post:
I don’t trust fallible human programmers to implement soundly “knowing more”, “thinking faster” and “growing up together”, and deal with the problems of “muddle”, “spread” and “distance”. The idea of a “last judge” as a safety measure seems like a sticking plaster on a gaping wound. Neither do I accept that including all of humanity is anything other than misplaced idealism. Some people seem to think that even a faulty CEV initial dynamic magically corrects itself into a good one; that might happen, but not with nearly a high enough probability.
Another problem that has been scarcely discussed: what happens if, as Eliezer’s CEV document suggests might happen, the thing shuts itself down or the last judge decides it isn’t safe? And the second time we try it, too?
But the problem remains that a superintelligence needs a full set of human values in order for it to be safe. I don’t see any other tenable proposals for implementing this apart from CEV, therefore I conclude that building a recursively improving superintelligence is basically just unsafe, given present human competence levels. Given that fact, to conclude that because we are likely to obtain the means to bring about a (positive or negative) singularity at some point we cannot prevent it from happening indefinitely is like saying that because we possess nuclear technology we can’t prevent a nuclear extinction event from happening indefinitely. If FAI is an “impossible” challenge and NAI (No AI) is merely very difficult, there is something to recommend NAI.
That doesn’t mean to say that I disprove of what Eliezer et al are doing. The singularity is definitely an extremely important thing to be discussing. I just think that the end product is likely to be widespread recognition of the peril of playing around with AI, and this (along with appropriately severe action taken to reduce the peril) is just as much a solution to Yudkowsky’s fear that a bunch of above-average AI scientists can “learn from each other’s partial successes and accumulate hacks as a community” as is trying to beat them to the punch by rushing to create a positive singularity.
Although this is unfair there is probably some truth to the idea that people who devote their lives to studying AI and the intelligence explosion are likely to be biased towards solutions to the problem in which their work achieves something really positive, rather than merely acting as a warning. That is not to pre-judge the issue, but merely to recommend that a little more skepticism than normal is due.
On the other hand there is another tenable approach to the singularity that is less widely recognised here. Wei Dai’s posts here and here seem very sensible to me; he suggests that intelligence enhancement should have priority over FAI research:
He quotes Eliezer as having said this (from pages 31-35 here):
Wei Dai points out that it is worth distinguishing the ease of creating uFAI in comparison to FAI, rather than lumping these together as “AI”.
I also think that the difference in outcomes between “deliberately not working on Friendly AI” and “treating unsupervised AI work as a terrible crime” are worth distinguishing.
This depends on the probability one assigns to CEV working. My probability that it would work given present human competence levels is low, and my probability that anyone would actually let it happen is very low.
The benefit of intelligence enhancement is that changes can be as unhurried and incremental as one likes (assuming that the risk of someone building uFAI is not considered to be imminent, due to stringent security measures); CEV is more a leap of faith.
Seconded—I’d like to see some material from lukeprog or somebody else at SI addressing these kinds of concerns. A “Criticisms of CEV” page maybe?
[Edit: just to clarify, I wasn’t seconding the part about the pitchforks and I’m not sure that either IA or an AGI ban is an obviously better strategy. But I agree with everything else here]
The problem is that for a nuclear explosion to take place, the higher ups of some country have to approve it.
For AGI, all that has to occur is code is leaked or hacked, then preventing the AGI from being implemented somewhere is an impossible task. And right now, no major online institution in the entire world is safe from being hacked.
Perhaps that is a motivation to completely (and effectively) prohibit research in the direction of creating superintelligent AI, licensed or otherwise, and concentrate entirely on human intelligence enhancement.
Ignoring how difficult this would be (due to ease of secret development considering it would largely consist of code rather than easy to track hardware) even if every country in the world WANTED to cooperate on it, the real problem comes from the high potential value of defecting. Much like nuclear weapons development, it would take a lot more than sanctions to convince a rogue nation NOT to try and develop AI for its own benefit, should this become a plausible course of action.
What would anyone think they stood to gain from creating an AI, if they understood the consequences as described by Yudkowsky et al?
The situation is not “much like nuclear weapons development”, because nuclear weapons are actually a practical warfare device, and the comparison was not intended to imply this similarity. I just meant to say that we manage to keep nukes out of the hands of terrorists, so there is reason to be optimistic about our chances of preventing irresponsible or crazy people from successfully developing a recursively self-improving AI—it is difficult, but if creating and successfully implementing a provably safe FAI (without prior intelligence enhancement) is hopelessly difficult—even if only because the large majority of people wouldn’t consent to it—then it may still be our best option.
The same things WE hope to gain from creating AI. I do not trust north korea (for example) to properly decide on the relative risks/rewards of any given course of action it can undertake.
OK but it isn’t hard (or wouldn’t be in the context we are discussing) to come to the understanding that creating an intelligence explosion renders the interests of any particular state irrelevant. I’ve seen no evidence that the North Koreans are that crazy.
The problem would be people who think that something like CEV, implemented by present-day humans, is actually safe—and the people liable to believe that are more likely to be the type of people found here, not North Koreans or other non-Westerners.
I’d also be interested in hearing your opinion on the security concerns should we attempt to implement CEV, and find that it shut itself down or produced an unacceptable output.
If you’re correct, then the best way to stave off the optimists from trying is to make an indisputable case for pessimism and disseminate it widely. Otherwise, eventually someone else will get optimistic, and won’t see why they shouldn’t give it a go.
I expect that once recognition of the intelligence explosion as a plausible scenario becomes mainstream, pessimism about the prospects of (unmodified) human programmers safely and successfully implementing CEV or some such thing will be the default, regardless of what certain AI researchers claim.
In that case, optimists are likely to have their activities forcefully curtailed. If this did not turn out to be the case, then I would consider “pro-pessimism” activism to change that state of affairs (assuming nothing happens to change my mind between now and then). At the moment however I support the activities of the Singularity Institute, because they are raising awareness of the problem (which is a prerequisite for state involvement) and they are highly responsible people. The worst state of affairs would be one in which no-one recognised the prospect of an intelligence explosion until it was too late.
ETA: I would be somewhat more supportive of a CEV in which only a select (and widely admired and recognised) group of humans was included. This seems to create an opportunity for the CEV initial dynamic implementation to be a compromise between intelligence enhancement and ordinary CEV, i.e. a small group of humans can be “prepared” and studied very carefully before the initial dynamic is switched on.
So really it’s a complex situation, and my post above probably failed to express the degree of ambivalence that I feel regarding this subject.
yeah, that’ll work.