“transparency”? I thought the point of your post was that SIAI members should refrain from making some of their beliefs easily available to the public?
I see, maybe I should have been more clear. The point of my post is that SIAI members should not express controversial views without substantiating them with abundant evidence. If SIAI provided compelling evidence that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing, then I would think Eliezer’s comment appropriate.
As things stand SIAI has not provided such evidence. Eliezer himself may have such evidence, but if so he’s either unwilling or unable to share it.
There are a lot of second and higher order effects in PR. You can always shape your public statements for one audience and end up driving away (or failing to convince) another one that’s more important. If Eliezer had shied away from stating some of the more “uncredible” ideas because there wasn’t enough evidence to convince a typical smart person, it would surely prompt questions of “what do you really think about this?” or fail to attract people who are currently interested in SIAI because of those ideas.
If SIAI provided compelling evidence that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing, then I would think Eliezer’s comment appropriate.
Suppose Eliezer hadn’t made that claim, and somebody asks him, “do you think the work SIAI is doing has higher expected value to humanity than what everybody else is doing?”, which somebody is bound to, given that Eliezer is asking for donations from rationalists. What is he supposed to say? “I can’t give you the answer because I don’t have enough evidence to convince a typical smart person?”
I think you make a good point that it’s important to think about PR, but I’m not at all convinced that the specific advice you give are the right ones.
You can always shape your public statements for one audience and end up driving away (or failing to convince) another one that’s more important.
This is of course true. I myself am fairly certain that SIAI’s public statements are driving away the people who it’s most important to interest in existential risk.
Suppose Eliezer hadn’t made that claim, and somebody asks him, “do you think the work SIAI is doing has higher expected value to humanity than what everybody else is doing?”, which somebody is bound to, given that Eliezer is asking for donations from rationalists. What is he supposed to say? “I can’t give you the answer because I don’t have enough evidence to convince a typical smart person?”
•It’s standard public relations practice to reveal certain information only if asked.
•An organization that has the strongest case for room for more funding need not be an organization that’s doing something of higher expected value to humanity than what everybody else is doing. In particular, I simultaneously believe that there are politicians who have higher expected value to humanity than all existential risk researchers alive and that the cause of existential risk has the greatest room for more funding.
•One need not be confident in one’s belief that funding one’s organization has highest expected value to humanity to believe that funding one’s organization has highest expected to humanity. A major issue that I have with Eliezer’s rhetoric is that he projects what I perceive to be an unreasonably high degree of confidence in his beliefs.
•Another major issue with Eliezer’s rhetoric that I have is that even putting issues of PR aside, I personally believe that funding SIAI does not have anywhere near the highest expected value to humanity out of all possible uses of money. So from my point of view, I see no upside to Eliezer making extreme claims of the sort that he has—it looks to me as though Eliezer is making false claims and damaging public relations for existential risk as a result.
I will be detailing my reasons for thinking that SIAI’s research does not have high expected value in a future post.
The level of certainty is not up for grabs. You are as confident as you happen to be, this can’t be changed. You can change the appearance, but not your actual level of confidence. And changing the apparent level of confidence is equivalent to lying.
But it isn’t perceived as so by the general public—it seems to me that the usual perception of “confidence” has more to do with status than with probability estimates.
The non-technical people I work with often say that I use “maybe” and “probably” too much (I’m a programmer—“it’ll probably work” is a good description of how often it does work in practice) - as if having confidence in one’s statements was a sign of moral fibre, and not a sign of miscalibration.
Actually, making statements with high confidence is a positive trait, but most people address this by increasing the confidence they express, not by increasing their knowledge until they can honestly make high-confidence statements. And our culture doesn’t correct for that, because errors of calibration are not immediatly obvious (as they would be if, say, we had a widespread habit of betting on various things).
higher expected value to humanity than what virtually everybody else is doing,
For what definitions of “value to humanity” and “virtually everybody else”?
If “value to humanity” is assessed as in Bostrom’s Astronomical Waste paper, that hugely favors effects on existential risk vs alleviating current suffering or increasing present welfare (as such, those also have existential risk effects). Most people don’t agree with that view, so asserting that as a privileged frame can be seen as a hostile move (attacking the value systems of others in favor of a value system according to which one’s area of focus is especially important). Think of the anger directed at vegetarians, or those who guilt-trip others about not saving African lives. And of course, it’s easier to do well on a metric that others are mostly not focused on optimizing.
Dispute about what best reduces existential risk, and annoyance at overly confident statements there is a further issue, but I think that asserting uncommon moral principles (which happen to rank one’s activities as much more valuable than most people would rank them) is a big factor on its own.
In case my previous comment was ambiguous, I should say that I agree with you completely on this point. I’ve been wanting to make a top level post about this general topic for a while. Not sure when I’ll get a chance to do so.
Eliezer himself may have such evidence [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing], but if so he’s either unwilling or unable to share it.
Now that is unfair.
Since 1997, Eliezer has published (mostly on mailing lists and blogs but also in monographs) an enormous amount (at least ten novels worth unless I am very mistaken) of writings supporting exactly that point. Of course most of this material is technical, but unlike the vast majority of technical prose, it is accessible to non-specialists and non-initiates with enough intelligence, a solid undergraduate education as a “scientific generalist” and a lot of free time on their hands because in his writings Eliezer is constantly “watching out for” the reader who does not yet know what he knows. (In other words, it is uncommonly good technical exposition.)
So my impression has been that the situation is that
(i) Eliezer’s writings contain a great deal of insightful material.
(ii) These writings do not substantiate the idea that [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing].
I say this having read perhaps around a thousand pages of what Eliezer has written. I consider the amount of reading that I’ve done to be a good “probabilistic proof” that the points (i) and (ii) apply to the portion of his writings that I haven’t read.
That being said, if there are any particular documents that you would point me to which you feel do provide a satisfactory evidence for the idea [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing], I would be happy to examine them.
I’m unwilling to read the whole of his opus given how much of it I’ve already read without being convinced. I feel that the time that I put into reducing existential risk can be used to better effect in other ways.
It would help to know what steps in the probabilistic proof don’t have high probability for you.
For example, you might think that the singularity has a good probability of being relatively smooth and some kind of friendly, even without FAI. or you might think that other existential risks may still be a bigger threat, or you may think that Eliezer isn’t putting a dent in the FAI problem.
This might be a convenient place to collect a variety of reasons why people are FOOM denialists. From my POV:
I am skeptical that safeguards against UFAI (unFAI) will not work. In part because:
I doubt that the “takeoff” will be “hard”. Because:
I am pretty sure the takeoff will require repeatedly doubling and quadrupling hardware, not just autorewriting software.
And hence an effective safeguard would be to simply not give the machine its own credit card!
And in any case, the Moore’s law curve for electronics does not arise from delays in thinking up clever ideas, it arises from delays in building machines to incredibly high tolerances.
Furthermore, even after the machine has more hardware, it doesn’t yet have higher intelligence until it reads lots more encyclopedias and proves for itself many more theorems. These things take time.
And finally, I have yet to see the argument that an FAI protects us from a future UFAI. That is, how does the SIAI help us?
Oh, and I do think that the other existential risks, particularly war and economic collapse, put the UFAI risk pretty far down the priority list. Sure, those other risks may not be quite so existential, but if they don’t kill us, they will at least prevent an early singularity.
Edit added two days later: Since writing this, I thought about it some more, shut up for a moment, and did the math. I still think that it is unlikely that the first takeoff will be a hard one; so hard that it gets out of control. But I now estimate something like a 10% chance that the first takeoff will be hard, and I estimate something like a 30% chance that at least one of the first couple dozen takeoffs will be hard. Multiply that by an estimated 10% chance that a hard takeoff will take place without adequate safeguards in place, and another 10% chance that a safeguardless hard takeoff will go rogue, and you get something like a 0.3% chance of a disaster of Forbin Project magnitude. Completely unacceptable.
Originally, I had discounted the chance that a simple software change could cause the takeoff; I assumed you would need to double and redouble the hardware capability. What I failed to notice was that a simple “tuning” change to the (soft) network connectivity parameters—changing the maximum number of inputs per “neuron” from
8 to 7, say, could have an (unexpected) effect on performance of several orders of magnitude simply by suppressing wasteful thrashing or some such thing.
I am pretty sure the takeoff will require repeatedly doubling and quadrupling hardware, not just autorewriting software.
Do you think that progress in AI is limited primarily by hardware? If hardware is the limiting factor, then you should think AI soon relatively plausible. If software is the limiting factor (the majority view, and the reason most AI folk reject claims such as those of Moravec), such that we won’t get AI until well beyond the minimum computational requirements, then either early AIs should be able to run fast or with numerous copies cheaply, or there will be a lot of room to reduce bloated hardware demands through software improvements.
Thinking that AI will take a long time (during which hardware will advance mightily towards physical limits) but also be sharply and stably hardware-limited when created is a hard view to defend.
I am imagining that it will work something like the human brain (but not by ‘scan and emulate’). We need to create hardware modules comparable to neurons, we need to have some kind of geometric organization which permits individual hardware modules to establish physical connections to a handful of nearby modules, and we need a ‘program’ (corresponding to human embryonic development) which establishes a few starting connections, and finally we need a training period (like training a neural net, and comparable to what the human brain experiences from the first neural activity in the womb through graduate school) which adds many more physical connections. I’m not sure whether to call these connections hardware or software. Actually, they are a hybrid of both—like PLAs (yeah, I way out of date on technology).
So I’m imagining a lot of theoretical work needed to come up with a good ‘neuron’ design (probably several dozen different kinds of neurons), more theoretical work to come up with a good ‘program’ to correspond to the embryonic interconnect, and someone willing to pay for lots and lots of neurons.
So, yeah, I’m thinking that the program will be relatively simple (equivalent to a few million lines of code), but it will take us a long time to find it. Not the 500 million years that it took evolution to come up with that program—apparently 500 million years after it had already invented the neuron. But for human designers, at least a few decades to find and write the program. I hope this explanation helps to make my position seem less weird.
4 . And hence an effective safeguard would be to simply not give the machine its own credit card!
(Powerful) optimization processes can find such ways of solving problems by exploiting every possible shortcut that it is hard to predict those ways in advance. Recently here was an example of that. Genetic algorithm found unexpected solution of a problem exploiting analog properties of particular FPGA chip.
3 and 4: hardware, sure—that is improving too—just not as fast, sometimes. A machine may find a way to obtain a credit card—or it will get a human to buy whatever it needs—as happens in companies today.
6: how much time? Surely a better example would be: “perform experiments”—and experiments that caan’t be minaturised and executed at high speeds—such as those done in the LHC.
7: AltaVista didn’t protect us from Google—nor did Friendster protect against MySpace. However, so far Google has mostly successfully crushed its rivals.
8: no way, IMO—e.g. see Matt Ridley. That is probably good advice for all DOOMsters, actually.
Some of the most obvious safeguards are likely to be self-imposed ones:
Can you be more specific than “it’s somewhere beneath an enormous amount of 13 years of material from the very same person whose arguments are scrutinized for evidence”?
This is not sufficient to scare people up to the point of having nightmares and ask them for most of their money.
Do you want me to repeat the links people gave you 24 hours ago?
The person who was scared to the point of having nightmares was almost certainly on a weeks-long or months-long visit to the big house in California where people come to discuss extremely powerful technologies and the far future and to learn from experts on these subjects. That environment would tend to cause a person to take certain ideas more seriously than a person usually would.
Also, are we really discrediting people because they were foolish enough to talk about their deranged sleep-thoughts? I’d sound pretty stupid too if I remembered and advertised every bit of nonsense I experienced while sleeping.
It was more than one person. Anyway, I haven’t read all of the comments yet so I might have missed some specific links. If you are talking about links to articles written by EY himself where he argues about AI going FOOM, I commented on one of them.
Here is an example of the kind of transparency in the form of strict calculations, references and evidence I expect.
As I said, I’m not sure what other links you are talking about. But if you mean the kind of LW posts dealing with antipredictions, I’m not impressed. Predicting superhuman AI to be a possible outcome of AI research is not sufficient. Where is the difference between claiming the LHC will go FOOM? I’m sure someone like EY would be able to write a thousand posts around such a scenario telling me that the high risk associated with the LHC going FOOM does outweigh its low probability. There might be sound arguments to support this conclusion. But it is a conclusion and a framework of arguments based on a assumption that is itself of unknown credibility. So is it too much to ask for some transparet evidence to fortify this basic premise? Evidence that is not somewhere to be found within hundreds of posts not directly concerned with the evidence in question but rather arguing based on the very assumption it is trying to justify?
Asteroids really are an easier problem: celestial mechanics in vacuum are pretty stable, we have the Moon providing a record of past cratering to calibrate on, etc. There’s still uncertainty about the technology of asteroid deflection (e.g. its potential for military use, or to incite conflict), but overall it’s perhaps the most tractable risk for analysis since the asteroids themselves don’t depend on recent events (save for some smallish anthropic shadow effects).
An analysis for engineered pathogens, where we have a lot of uncertainty about the difficulty of engineering various of diseases for maximum damage, and how the technology for detection, treatment and prevention will keep pace. We can make generalizations based on existing diseases and their evolutionary dynamics (selection for lower virulence over time with person-to-person transmission, etc), current public health measures, etc, the rarity of the relevant motivations, etc, but you’re still left with many more places where you can’t just plug in well-established numbers and crank forward.
You can still give probability estimates, and plug in well-understood past data where you can, but you can’t get asteroid-level exactitude.
The difference is that we understand both asteroids and particle physics far better than we do intelligence, and there is precedence for both asteroid impacts and high energy particle collisions (natural ones at far higher energy than in the LHC) while there is none for an engineered human level intelligence with access to its own source code.
So calculations of the kind you seem to be asking for just aren’t possible at this point (and calculations with exactly that level of evidence won’t be possible right up until it’s too late), while refutations of the kind LHC panic gets aren’t possible either. You should also note that Eliezer takes LHC panic more serious than most non-innumerate people.
But if you want some calculation anyway: Let’s assume there is a 1% chance of extinction by uFAI within the next 100 years. Let’s also assume that spending $10 million per year (in 2010 dollars, adjusting for inflation) allows us to reduce that risk by 10%, just by the dangers of uFAI being in the public eye and people being somewhat more cautious, and taking the right sort of caution instead of worrying about Skynet or homicidal robots. So $1 billion saves about an expected 1 million lives, a cost of $ 1000 per life, which is about the level of the most efficient conventional charities. And that’s with Robins low-balling estimate (which was for a more specific case, not uFAI extinction in general, so even Robin would likely estimate a higher chance in the case considered) and assuming that FAI research won’t succeed.
So calculations of the kind you seem to be asking for just aren’t possible at this point …
I’m asking for whatever calculations should lead people to donate most of their money to the SIAI or get nightmares from stories of distant FAI’s. Surely there must be something to outweigh the lack of evidence, or on what basis has anyone decided to take things serious?
I really don’t want to anger you but the “let’s assume X” attitude is what I have my problems with here. A 1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
I wonder why it took so long until you came along posting this comment.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
Have you yet bothered to read e.g. this synopsis of SIAI’s position:
“Many AIs will converge toward being optimizing systems, in the sense that, after self-modification, they will act to maximize some goal. For instance, AIs developed under evolutionary pressures would be selected for values that maximized reproductive fitness, and would prefer to allocate resources to reproduction rather than supporting humans. Such unsafe AIs might actively mimic safe benevolence until they became powerful, since being destroyed would prevent them from working toward their goals. Thus, a broad range of AI designs may initially appear safe, but if developed to the point of a Singularity could cause human extinction in the course of optimizing the Earth for their goals.”
Personally, I think that presents a very weak case for there being risk. It argues that there could be risk if we built these machines wrong, and the bad machines became powerful somehow. That is true—but the reader is inclined to respond “so what”. A dam can be dangerous if you build it wrong too. Such observations don’t say very much about the actual risk.
I am very sceptical about that being true for those alive now:
We have been looking for things that might hit us for a long while now—and we can see much more clearly what the chances are for that period than by looking at the historical record. Also, that is apparently assuming no mitigation attempts—which also seems totally unrealistic.
...gives 700 deaths/year for aircraft—and 1,400 deaths/year for 2km impacts—based on assumption that one quarter of the human population would perish in such an impact.
“transparency”? I thought the point of your post was that SIAI members should refrain from making some of their beliefs easily available to the public?
I see, maybe I should have been more clear. The point of my post is that SIAI members should not express controversial views without substantiating them with abundant evidence. If SIAI provided compelling evidence that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing, then I would think Eliezer’s comment appropriate.
As things stand SIAI has not provided such evidence. Eliezer himself may have such evidence, but if so he’s either unwilling or unable to share it.
There are a lot of second and higher order effects in PR. You can always shape your public statements for one audience and end up driving away (or failing to convince) another one that’s more important. If Eliezer had shied away from stating some of the more “uncredible” ideas because there wasn’t enough evidence to convince a typical smart person, it would surely prompt questions of “what do you really think about this?” or fail to attract people who are currently interested in SIAI because of those ideas.
Suppose Eliezer hadn’t made that claim, and somebody asks him, “do you think the work SIAI is doing has higher expected value to humanity than what everybody else is doing?”, which somebody is bound to, given that Eliezer is asking for donations from rationalists. What is he supposed to say? “I can’t give you the answer because I don’t have enough evidence to convince a typical smart person?”
I think you make a good point that it’s important to think about PR, but I’m not at all convinced that the specific advice you give are the right ones.
Thanks for your feedback. Several remarks:
This is of course true. I myself am fairly certain that SIAI’s public statements are driving away the people who it’s most important to interest in existential risk.
•It’s standard public relations practice to reveal certain information only if asked.
•An organization that has the strongest case for room for more funding need not be an organization that’s doing something of higher expected value to humanity than what everybody else is doing. In particular, I simultaneously believe that there are politicians who have higher expected value to humanity than all existential risk researchers alive and that the cause of existential risk has the greatest room for more funding.
•One need not be confident in one’s belief that funding one’s organization has highest expected value to humanity to believe that funding one’s organization has highest expected to humanity. A major issue that I have with Eliezer’s rhetoric is that he projects what I perceive to be an unreasonably high degree of confidence in his beliefs.
•Another major issue with Eliezer’s rhetoric that I have is that even putting issues of PR aside, I personally believe that funding SIAI does not have anywhere near the highest expected value to humanity out of all possible uses of money. So from my point of view, I see no upside to Eliezer making extreme claims of the sort that he has—it looks to me as though Eliezer is making false claims and damaging public relations for existential risk as a result.
I will be detailing my reasons for thinking that SIAI’s research does not have high expected value in a future post.
The level of certainty is not up for grabs. You are as confident as you happen to be, this can’t be changed. You can change the appearance, but not your actual level of confidence. And changing the apparent level of confidence is equivalent to lying.
But it isn’t perceived as so by the general public—it seems to me that the usual perception of “confidence” has more to do with status than with probability estimates.
The non-technical people I work with often say that I use “maybe” and “probably” too much (I’m a programmer—“it’ll probably work” is a good description of how often it does work in practice) - as if having confidence in one’s statements was a sign of moral fibre, and not a sign of miscalibration.
Actually, making statements with high confidence is a positive trait, but most people address this by increasing the confidence they express, not by increasing their knowledge until they can honestly make high-confidence statements. And our culture doesn’t correct for that, because errors of calibration are not immediatly obvious (as they would be if, say, we had a widespread habit of betting on various things).
That a lie is likely to be misinterpreted or not noticed doesn’t make it not a lie, and conversely.
Oh, I fully agree with your point; it’s a pity that high confidence on unusual topics is interpreted as arrogance.
Try this: I prefer my leaders to be confident. I prefer my subordinates to be truthful.
For what definitions of “value to humanity” and “virtually everybody else”?
If “value to humanity” is assessed as in Bostrom’s Astronomical Waste paper, that hugely favors effects on existential risk vs alleviating current suffering or increasing present welfare (as such, those also have existential risk effects). Most people don’t agree with that view, so asserting that as a privileged frame can be seen as a hostile move (attacking the value systems of others in favor of a value system according to which one’s area of focus is especially important). Think of the anger directed at vegetarians, or those who guilt-trip others about not saving African lives. And of course, it’s easier to do well on a metric that others are mostly not focused on optimizing.
Dispute about what best reduces existential risk, and annoyance at overly confident statements there is a further issue, but I think that asserting uncommon moral principles (which happen to rank one’s activities as much more valuable than most people would rank them) is a big factor on its own.
In case my previous comment was ambiguous, I should say that I agree with you completely on this point. I’ve been wanting to make a top level post about this general topic for a while. Not sure when I’ll get a chance to do so.
Now that is unfair.
Since 1997, Eliezer has published (mostly on mailing lists and blogs but also in monographs) an enormous amount (at least ten novels worth unless I am very mistaken) of writings supporting exactly that point. Of course most of this material is technical, but unlike the vast majority of technical prose, it is accessible to non-specialists and non-initiates with enough intelligence, a solid undergraduate education as a “scientific generalist” and a lot of free time on their hands because in his writings Eliezer is constantly “watching out for” the reader who does not yet know what he knows. (In other words, it is uncommonly good technical exposition.)
So my impression has been that the situation is that
(i) Eliezer’s writings contain a great deal of insightful material.
(ii) These writings do not substantiate the idea that [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing].
I say this having read perhaps around a thousand pages of what Eliezer has written. I consider the amount of reading that I’ve done to be a good “probabilistic proof” that the points (i) and (ii) apply to the portion of his writings that I haven’t read.
That being said, if there are any particular documents that you would point me to which you feel do provide a satisfactory evidence for the idea [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing], I would be happy to examine them.
I’m unwilling to read the whole of his opus given how much of it I’ve already read without being convinced. I feel that the time that I put into reducing existential risk can be used to better effect in other ways.
It would help to know what steps in the probabilistic proof don’t have high probability for you.
For example, you might think that the singularity has a good probability of being relatively smooth and some kind of friendly, even without FAI. or you might think that other existential risks may still be a bigger threat, or you may think that Eliezer isn’t putting a dent in the FAI problem.
Or some combination of these and others.
Yes, I agree with you. I plan on making my detailed thoughts on these points explicit. I expect to be able to do so within a month.
But for a short answer, I would say that the situation is mostly that I think that:
This might be a convenient place to collect a variety of reasons why people are FOOM denialists. From my POV:
I am skeptical that safeguards against UFAI (unFAI) will not work. In part because:
I doubt that the “takeoff” will be “hard”. Because:
I am pretty sure the takeoff will require repeatedly doubling and quadrupling hardware, not just autorewriting software.
And hence an effective safeguard would be to simply not give the machine its own credit card!
And in any case, the Moore’s law curve for electronics does not arise from delays in thinking up clever ideas, it arises from delays in building machines to incredibly high tolerances.
Furthermore, even after the machine has more hardware, it doesn’t yet have higher intelligence until it reads lots more encyclopedias and proves for itself many more theorems. These things take time.
And finally, I have yet to see the argument that an FAI protects us from a future UFAI. That is, how does the SIAI help us?
Oh, and I do think that the other existential risks, particularly war and economic collapse, put the UFAI risk pretty far down the priority list. Sure, those other risks may not be quite so existential, but if they don’t kill us, they will at least prevent an early singularity.
Edit added two days later: Since writing this, I thought about it some more, shut up for a moment, and did the math. I still think that it is unlikely that the first takeoff will be a hard one; so hard that it gets out of control. But I now estimate something like a 10% chance that the first takeoff will be hard, and I estimate something like a 30% chance that at least one of the first couple dozen takeoffs will be hard. Multiply that by an estimated 10% chance that a hard takeoff will take place without adequate safeguards in place, and another 10% chance that a safeguardless hard takeoff will go rogue, and you get something like a 0.3% chance of a disaster of Forbin Project magnitude. Completely unacceptable.
Originally, I had discounted the chance that a simple software change could cause the takeoff; I assumed you would need to double and redouble the hardware capability. What I failed to notice was that a simple “tuning” change to the (soft) network connectivity parameters—changing the maximum number of inputs per “neuron” from 8 to 7, say, could have an (unexpected) effect on performance of several orders of magnitude simply by suppressing wasteful thrashing or some such thing.
Do you think that progress in AI is limited primarily by hardware? If hardware is the limiting factor, then you should think AI soon relatively plausible. If software is the limiting factor (the majority view, and the reason most AI folk reject claims such as those of Moravec), such that we won’t get AI until well beyond the minimum computational requirements, then either early AIs should be able to run fast or with numerous copies cheaply, or there will be a lot of room to reduce bloated hardware demands through software improvements.
Thinking that AI will take a long time (during which hardware will advance mightily towards physical limits) but also be sharply and stably hardware-limited when created is a hard view to defend.
I am imagining that it will work something like the human brain (but not by ‘scan and emulate’). We need to create hardware modules comparable to neurons, we need to have some kind of geometric organization which permits individual hardware modules to establish physical connections to a handful of nearby modules, and we need a ‘program’ (corresponding to human embryonic development) which establishes a few starting connections, and finally we need a training period (like training a neural net, and comparable to what the human brain experiences from the first neural activity in the womb through graduate school) which adds many more physical connections. I’m not sure whether to call these connections hardware or software. Actually, they are a hybrid of both—like PLAs (yeah, I way out of date on technology).
So I’m imagining a lot of theoretical work needed to come up with a good ‘neuron’ design (probably several dozen different kinds of neurons), more theoretical work to come up with a good ‘program’ to correspond to the embryonic interconnect, and someone willing to pay for lots and lots of neurons.
So, yeah, I’m thinking that the program will be relatively simple (equivalent to a few million lines of code), but it will take us a long time to find it. Not the 500 million years that it took evolution to come up with that program—apparently 500 million years after it had already invented the neuron. But for human designers, at least a few decades to find and write the program. I hope this explanation helps to make my position seem less weird.
(Powerful) optimization processes can find such ways of solving problems by exploiting every possible shortcut that it is hard to predict those ways in advance. Recently here was an example of that. Genetic algorithm found unexpected solution of a problem exploiting analog properties of particular FPGA chip.
7-8 aren’t hard-takeoff-denialist ideas; they’re SIAI noncontribution arguments. Good summary, though.
Phew! First, my material on the topic:
http://alife.co.uk/essays/the_singularity_is_nonsense/
http://alife.co.uk/essays/the_intelligence_explosion_is_happening_now/
Then a few points—which I may add to later.
3 and 4: hardware, sure—that is improving too—just not as fast, sometimes. A machine may find a way to obtain a credit card—or it will get a human to buy whatever it needs—as happens in companies today.
6: how much time? Surely a better example would be: “perform experiments”—and experiments that caan’t be minaturised and executed at high speeds—such as those done in the LHC.
7: AltaVista didn’t protect us from Google—nor did Friendster protect against MySpace. However, so far Google has mostly successfully crushed its rivals.
8: no way, IMO—e.g. see Matt Ridley. That is probably good advice for all DOOMsters, actually.
Some of the most obvious safeguards are likely to be self-imposed ones:
http://alife.co.uk/essays/stopping_superintelligence/
...though a resiliant infrastructure would help too. We see rogue agents (botnets) “eating” the internet today—and it is not very much fun!
Incidentally, a much better place for this kind of comment on this site would be:
http://lesswrong.com/lw/wf/hard_takeoff/
Can you be more specific than “it’s somewhere beneath an enormous amount of 13 years of material from the very same person whose arguments are scrutinized for evidence”?
This is not sufficient to scare people up to the point of having nightmares and ask them for most of their money.
Do you want me to repeat the links people gave you 24 hours ago?
The person who was scared to the point of having nightmares was almost certainly on a weeks-long or months-long visit to the big house in California where people come to discuss extremely powerful technologies and the far future and to learn from experts on these subjects. That environment would tend to cause a person to take certain ideas more seriously than a person usually would.
Also, are we really discrediting people because they were foolish enough to talk about their deranged sleep-thoughts? I’d sound pretty stupid too if I remembered and advertised every bit of nonsense I experienced while sleeping.
It was more than one person. Anyway, I haven’t read all of the comments yet so I might have missed some specific links. If you are talking about links to articles written by EY himself where he argues about AI going FOOM, I commented on one of them.
Here is an example of the kind of transparency in the form of strict calculations, references and evidence I expect.
As I said, I’m not sure what other links you are talking about. But if you mean the kind of LW posts dealing with antipredictions, I’m not impressed. Predicting superhuman AI to be a possible outcome of AI research is not sufficient. Where is the difference between claiming the LHC will go FOOM? I’m sure someone like EY would be able to write a thousand posts around such a scenario telling me that the high risk associated with the LHC going FOOM does outweigh its low probability. There might be sound arguments to support this conclusion. But it is a conclusion and a framework of arguments based on a assumption that is itself of unknown credibility. So is it too much to ask for some transparet evidence to fortify this basic premise? Evidence that is not somewhere to be found within hundreds of posts not directly concerned with the evidence in question but rather arguing based on the very assumption it is trying to justify?
Asteroids really are an easier problem: celestial mechanics in vacuum are pretty stable, we have the Moon providing a record of past cratering to calibrate on, etc. There’s still uncertainty about the technology of asteroid deflection (e.g. its potential for military use, or to incite conflict), but overall it’s perhaps the most tractable risk for analysis since the asteroids themselves don’t depend on recent events (save for some smallish anthropic shadow effects).
An analysis for engineered pathogens, where we have a lot of uncertainty about the difficulty of engineering various of diseases for maximum damage, and how the technology for detection, treatment and prevention will keep pace. We can make generalizations based on existing diseases and their evolutionary dynamics (selection for lower virulence over time with person-to-person transmission, etc), current public health measures, etc, the rarity of the relevant motivations, etc, but you’re still left with many more places where you can’t just plug in well-established numbers and crank forward.
You can still give probability estimates, and plug in well-understood past data where you can, but you can’t get asteroid-level exactitude.
The difference is that we understand both asteroids and particle physics far better than we do intelligence, and there is precedence for both asteroid impacts and high energy particle collisions (natural ones at far higher energy than in the LHC) while there is none for an engineered human level intelligence with access to its own source code.
So calculations of the kind you seem to be asking for just aren’t possible at this point (and calculations with exactly that level of evidence won’t be possible right up until it’s too late), while refutations of the kind LHC panic gets aren’t possible either. You should also note that Eliezer takes LHC panic more serious than most non-innumerate people.
But if you want some calculation anyway: Let’s assume there is a 1% chance of extinction by uFAI within the next 100 years. Let’s also assume that spending $10 million per year (in 2010 dollars, adjusting for inflation) allows us to reduce that risk by 10%, just by the dangers of uFAI being in the public eye and people being somewhat more cautious, and taking the right sort of caution instead of worrying about Skynet or homicidal robots. So $1 billion saves about an expected 1 million lives, a cost of $ 1000 per life, which is about the level of the most efficient conventional charities. And that’s with Robins low-balling estimate (which was for a more specific case, not uFAI extinction in general, so even Robin would likely estimate a higher chance in the case considered) and assuming that FAI research won’t succeed.
I’m asking for whatever calculations should lead people to donate most of their money to the SIAI or get nightmares from stories of distant FAI’s. Surely there must be something to outweigh the lack of evidence, or on what basis has anyone decided to take things serious?
I really don’t want to anger you but the “let’s assume X” attitude is what I have my problems with here. A 1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
So what’s the alternative?
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
This isn’t quite what you’re talking about, but a relatively accessible intro doc:
http://singinst.org/riskintro/index.html
This seems like a summary of the idea of there being significant risk:
Anna Salamon at Singularity Summit 2009 - “Shaping the Intelligence Explosion”
http://www.vimeo.com/7318055
Good comment.
However,
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
Have you yet bothered to read e.g. this synopsis of SIAI’s position:
http://singinst.org/riskintro/index.html
I’d also strongly recommend this from Bostrom:
http://www.nickbostrom.com/fut/evolution.html
(Then of course there are longer and more comprehensive texts, which I won’t recommend because you would just continue to ignore them.)
The core of:
http://singinst.org/riskintro/
...that talks about risk appears to be:
“Many AIs will converge toward being optimizing systems, in the sense that, after self-modification, they will act to maximize some goal. For instance, AIs developed under evolutionary pressures would be selected for values that maximized reproductive fitness, and would prefer to allocate resources to reproduction rather than supporting humans. Such unsafe AIs might actively mimic safe benevolence until they became powerful, since being destroyed would prevent them from working toward their goals. Thus, a broad range of AI designs may initially appear safe, but if developed to the point of a Singularity could cause human extinction in the course of optimizing the Earth for their goals.”
Personally, I think that presents a very weak case for there being risk. It argues that there could be risk if we built these machines wrong, and the bad machines became powerful somehow. That is true—but the reader is inclined to respond “so what”. A dam can be dangerous if you build it wrong too. Such observations don’t say very much about the actual risk.
This calculation places no value on the future generations whose birth depends on averting existential risk. That’s not how I see things.
That claims that “that the lifetime risk of dying from an asteroid strike is about the same as the risk of dying in a commercial airplane crash”.
It cites:
Impacts on the Earth by asteroids and comets: assessing the hazard:
http://www.nature.com/nature/journal/v367/n6458/abs/367033a0.html
I am very sceptical about that being true for those alive now:
We have been looking for things that might hit us for a long while now—and we can see much more clearly what the chances are for that period than by looking at the historical record. Also, that is apparently assuming no mitigation attempts—which also seems totally unrealistic.
Looking further:
http://users.tpg.com.au/users/tps-seti/spacegd7.html
...gives 700 deaths/year for aircraft—and 1,400 deaths/year for 2km impacts—based on assumption that one quarter of the human population would perish in such an impact.
Yet, does the SIAI provide evidence on par with the paper I linked to?
What—about the chances of superintelligence causing THE END OF THE WORLD?!?
Of course not! How could they be expected to do that?
If there really was “abundant evidence” there probably wouldn’t be much of a controversy.