Risks from AI and Charitable Giving
If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments. But if you’re interested in producing truth, you will fix your opponents’ arguments for them. To win, you must fight not only the creature you encounter; you [also] must fight the most horrible thing that can be constructed from its corpse.
This is an informal post meant as a reply to a post by user:utilitymonster, ‘What is the best compact formalization of the argument for AI risk from fast takeoff?’
I hope to find the mental strength to put more effort into it in future to improve it. But since nobody else seems to be willing to take a critical look at the overall topic I feel that doing what I can is better than doing nothing.
Please review the categories ‘Further Reading’ and ‘Notes and References’.
Contents
Abstract
In this post I just want to take a look at a few premises (P#) that need to be true simultaneously to make the SIAI a wortwhile charity from the point of view of someone trying to do as much good as possible by contributing money. I am going to show that the case of risks from AI is strongly conjunctive, that without a concrete and grounded understanding of AGI an abstract analysis of the issues is going to be very shaky, and that therefore SIAI is likely to be a bad choice as a charity. In other words, that which speaks in favor of SIAI does mainly consist of highly specific, conjunctive, non-evidence-backed speculations on possible bad outcomes.
Requirements for an Intelligence Explosion
P1 Fast, and therefore dangerous, recursive self-improvement is logically possible.
It took almost four hundred years to prove Fermat’s Last Theorem. The final proof is over a hundred pages long. Over a hundred pages! And we are not talking about something like an artificial general intelligence that can magically make itself smart enough to prove such theorems and many more that no human being would be capable of proving. Fermat’s Last Theorem simply states “no three positive integers a, b, and c can satisfy the equation a^n + b^n = c^n for any integer value of n greater than two.”
Even artificial intelligence researchers admit that “there could be non-linear complexity constrains meaning that even theoretically optimal algorithms experience strongly diminishing intelligence returns for additional compute power.” [1] We just don’t know.
Other possible problems include the impossibility of a stable utility function and a reflective decision theory, the intractability of real world expected utility maximization or that expected utility maximizers stumble over Pascal’s mugging, among other things [2].
For an AI to be capable of recursive self-improvement it also has to guarantee that its goals will be preserved when it improves itself. It is still questionable if it is possible to conclusively prove that improvements to an agent’s intelligence or decision procedures maximize expected utility. If this isn’t possible it won’t be rational or possible to undergo explosive self-improvement.
P1.b The fast computation of a simple algorithm is sufficient to outsmart and overpower humanity.
Imagine a group of 100 world-renowned scientists and military strategists.
The group is analogous to the initial resources of an AI.
The knowledge that the group has is analogous to what an AI could come up with by simply “thinking” about it given its current resources.
Could such a group easily wipe away the Roman empire when beamed back in time?
The Roman empire is analogous to our society today.
Even if you gave all of them a machine gun, the Romans would quickly adapt and the people from the future would run out of ammunition.
Machine guns are analogous to the supercomputer it runs on.
Consider that it takes a whole technological civilization to produce a modern smartphone.
You can’t just say “with more processing power you can do more different things”, that would be analogous to saying that “100 people” from today could just build more “machine guns”. But they can’t! They can’t use all their knowledge and magic from the future to defeat the Roman empire.
A lot of assumptions have to turn out to be correct to make humans discover simple algorithms over night that can then be improved to self-improve explosively.
You can also compare this to the idea of a Babylonian mathematician discovering modern science and physics given that he would be uploaded into a supercomputer (a possibility that is in and of itself already highly speculative). It assumes that he could brute-force conceptual revolutions.
Even if he was given a detailed explanation of how his mind works and the resources to understand it, self-improving to achieve superhuman intelligence assumes that throwing resources at the problem of intelligence will magically allow him to pull improved algorithms from solution space as if they were signposted.
But unknown unknowns are not signposted. It’s rather like finding a needle in a haystack. Evolution is great at doing that and assuming that one could speed up evolution considerably is another assumption about technological feasibility and real-world resources.
That conceptual revolutions are just a matter of computational resources is pure speculation.
If one were to speed up the whole Babylonian world and accelerate cultural evolution, obviously one would arrive quicker at some insights. But how much quicker? How much are many insights dependent on experiments, to yield empirical evidence, that can’t be speed-up considerably? And what is the return? Is the payoff proportionally to the resources that are necessary?
If you were going to speed up a chimp brain a million times, would it quickly reach human-level intelligence? If not, why then would it be different for a human-level intelligence trying to reach transhuman intelligence? It seems like a nice idea when formulated in English, but would it work?
Being able to state that an AI could use some magic to take over the earth does not make it a serious possibility.
Magic has to be discovered, adapted and manufactured first. It doesn’t just emerge out of nowhere from the computation of certain algorithms. It emerges from a society of agents with various different goals and heuristics like “Treating Rare Diseases in Cute Kittens”. It is an evolutionary process that relies on massive amounts of real-world feedback and empirical experimentation. Assuming that all that can happen because some simple algorithm is being computed is like believing it will emerge ‘out of nowhere’, it is magical thinking.
Unknown unknowns are not sign-posted. [3]
If people like Benoît B. Mandelbrot would have never decided to research Fractals then many modern movies wouldn’t be possible, as they rely on fractal landscape algorithms. Yet, at the time Benoît B. Mandelbrot conducted his research it was not foreseeable that his work would have any real-world applications.
Important discoveries are made because many routes with low or no expected utility are explored at the same time [4]. And to do so efficiently it takes random mutation, a whole society of minds, a lot of feedback and empirical experimentation.
“Treating rare diseases in cute kittens” might or might not provide genuine insights and open up new avenues for further research. As long as you don’t try it you won’t know.
The idea that a rigid consequentialist with simple values can think up insights and conceptual revolutions simply because it is instrumentally useful to do so is implausible.
Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution.
You need to have various different agents with different utility-functions around to get the necessary diversity that can give rise to enough selection pressure. A “singleton” won’t be able to predict the actions of new and improved versions of itself by just running sandboxed simulations. Not just because of logical uncertainty but also because it is computationally intractable to predict the real-world payoff of changes to its decision procedures.
You need complex values to give rise to the necessary drives to function in a complex world. You can’t just tell an AI to protect itself. What would that even mean? What changes are illegitimate? What constitutes “self”? That are all unsolved problems that are just assumed to be solvable when talking about risks from AI.
An AI with simple values will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue. Which will allow an AI to solve some well-defined narrow problems, but it will be unable to make use of the broad range of synergetic effects of cultural evolution. Cultural evolution is a result of the interaction of a wide range of utility-functions.
Yet even if we assume that there is one complete theory of general intelligence, once discovered, one just has to throw more resources at it. It might be able to incorporate all human knowledge, adapt it and find new patterns. But would it really be vastly superior to human society and their expert systems?
Can intelligence itself be improved apart from solving well-defined problems and making more accurate predictions on well-defined classes of problems? The discovery of unknown unknowns does not seem to be subject to other heuristics than natural selection. Without goals, well-defined goals, terms like “optimization” have no meaning.
P2 Fast, and therefore dangerous, recursive self-improvement is physically possible.
Even if it could be proven that explosive recursive self-improvement is logically possible, e.g. that there are no complexity constraints, the question remains if it is physically possible.
Our best theories about intelligence are highly abstract and their relation to real world human-level general intelligence is often wildly speculative [5][6].
P3 Fast, and therefore dangerous, recursive self-improvement is economically feasible.
To exemplify the problem take the science fictional idea of using antimatter as explosive for weapons. It is physically possible to produce antimatter and use it for large scale destruction. An equivalent of the Hiroshima atomic bomb will only take half a gram of antimatter. But it will take 2 billion years to produce that amount of antimatter [7].
We simply don’t know if intelligence is instrumental or quickly hits diminishing returns [8].
P3.b AGI is able to create (or acquire) resources, empowering technologies or civilisatory support [9].
We are already at a point where we have to build billion dollar chip manufacturing facilities to run our mobile phones. We need to build huge particle accelerators to obtain new insights into the nature of reality.
An AI would either have to rely on the help of a whole technological civilization or be in control of advanced nanotech assemblers.
And if an AI was to acquire the necessary resources on its own, its plan for world-domination would have to go unnoticed. This would require the workings of the AI to be opaque to its creators yet comprehensible to itself.
But an AI capable of efficient recursive self improvement must be able to
comprehend its own workings
predict how improvements, respectively improved versions of itself, are going to act to ensure that its values are preserved
So if the AI can do that, why wouldn’t humans be able to use the same algorithms to predict what the initial AI is going to do? And if the AI can’t do that, how is it going to maximize expected utility if it is unable to predict what it is going to do?
Any AI capable of efficient self-modification must be able to grasp its own workings and make predictions about improvements to various algorithms and its overall decision procedure. If an AI can do that, why would the humans who build it be unable to notice any malicious intentions? Why wouldn’t the humans who created it not be able to use the same algorithms that the AI uses to predict what it will do? If humans are unable to predict what the AI will do, how is the AI able to predict what improved versions of itself will do?
And even if an AI was able to somehow acquire large amounts of money. It is not easy to use the money. You can’t “just” build huge companies with fake identities, or a straw man, to create revolutionary technologies easily. Running companies with real people takes a lot of real-world knowledge, interactions and feedback. But most importantly, it takes a lot of time. An AI could not simply create a new Intel or Apple over a few years without its creators noticing anything.
The goals of an AI will be under scrutiny at any time. It seems very implausible that scientists, a company or the military are going to create an AI and then just let it run without bothering about its plans. An artificial agent is not a black box, like humans are, where one is only able to guess its real intentions.
A plan for world domination seems like something that can’t be concealed from its creators. Lying is no option if your algorithms are open to inspection.
P4 Dangerous recursive self-improvement is the default outcome of the creation of artificial general intelligence.
Complex goals need complex optimization parameters (the design specifications of the subject of the optimization process against which it will measure its success of self-improvement).
Even the creation of paperclips is a much more complex goal than telling an AI to compute as many decimal digits of Pi as possible.
For an AGI, that was designed to design paperclips, to pose an existential risk, its creators would have to be capable enough to enable it to take over the universe on its own, yet forget, or fail to, define time, space and energy bounds as part of its optimization parameters. Therefore, given the large amount of restrictions that are inevitably part of any advanced general intelligence (AGI), the nonhazardous subset of all possible outcomes might be much larger than that where the AGI works perfectly yet fails to hold before it could wreak havoc.
And even given a rational utility maximizer. It is possible to maximize paperclips in a lot of different ways. How it does it is fundamentally dependent on its utility-function and how precisely it was defined.
If there are no constraints in the form of design and goal parameters then it can maximize paperclips in all sorts of ways that don’t demand recursive self-improvement.
“Utility” does only become well-defined if we precisely define what it means to maximize it. Just maximizing paperclips doesn’t define how quickly and how economically it is supposed to happen.
The problem is that “utility” has to be defined. To maximize expected utility does not imply certain actions, efficiency and economic behavior, or the drive to protect yourself. You can also rationally maximize paperclips without protecting yourself if it is not part of your goal parameters.
You can also assign utility to maximize paperclips as long as nothing turns you off but don’t care about being turned off. If an AI is not explicitly programmed to care about it, then it won’t.
Without well-defined goals in form of a precise utility-function, it might be impossible to maximize expected “utility”. Concepts like “efficient”, “economic” or “self-protection” all have a meaning that is inseparable with an agent’s terminal goals. If you just tell it to maximize paperclips then this can be realized in an infinite number of ways that would all be rational given imprecise design and goal parameters. Undergoing to explosive recursive self-improvement, taking over the universe and filling it with paperclips, is just one outcome. Why would an arbitrary mind pulled from mind-design space care to do that? Why not just wait for paperclips to arise due to random fluctuations out of a state of chaos? That wouldn’t be irrational. To have an AI take over the universe as fast as possible you would have to explicitly design it to do so.
But for the sake of a thought experiment assume that the default case was recursive self-improvement. Now imagine that a company like Apple wanted to build an AI that could answer every question (an Oracle).
If Apple was going to build an Oracle it would anticipate that other people would also want to ask it questions. Therefore it can’t just waste all resources on looking for an inconsistency arising from the Peano axioms when asked to solve 1+1. It would not devote additional resources on answering those questions that are already known to be correct with a high probability. It wouldn’t be economically useful to take over the universe to answer simple questions.
It would neither be rational to look for an inconsistency arising from the Peano axioms while solving 1+1. To answer questions an Oracle needs a good amount of general intelligence. And concluding that asking it to solve 1+1 implies to look for an inconsistency arising from the Peano axioms does not seem reasonable. It also does not seem reasonable to suspect that humans desire an answer to their questions to approach infinite certainty. Why would someone build such an Oracle in the first place?
A reasonable Oracle would quickly yield good solutions by trying to find answers within a reasonable time which are with a high probability just 2–3% away from the optimal solution. I don’t think anyone would build an answering machine that throws the whole universe at the first sub-problem it encounters.
P5 The human development of artificial general intelligence will take place quickly.
What evidence do we have that there is some principle that, once discovered, allows us to grow superhuman intelligence overnight?
If the development of AGI takes place slowly, a gradual and controllable development, we might be able to learn from small-scale mistakes, or have enough time to develop friendly AI, while having to face other existential risks.
This might for example be the case if intelligence can not be captured by a discrete algorithm, or is modular, and therefore never allow us to reach a point where we can suddenly build the smartest thing ever that does just extend itself indefinitely.
Therefore the probability of an AI to undergo explosive recursive self-improvement (P(FOOM)) is the probability of the conjunction (P#∧P#) of its premises:
P(FOOM) = P(P1∧P2∧P3∧P4∧P5)
Of course, there are many more premises that need to be true in order to enable an AI to go FOOM, e.g. that each level of intelligence can effectively handle its own complexity, or that most AGI designs can somehow self-modify their way up to massive superhuman intelligence. But I believe that the above points are enough to show that the case for a hard takeoff is not disjunctive, but rather strongly conjunctive.
Requirements for SIAI to constitute an optimal charity
In this section I will assume the truth of all premises in the previous section.
P6 SIAI can solve friendly AI.
Say you believe that unfriendly AI will wipe us out with a probability of 60% and that there is another existential risk that will wipe us out with a probability of 10% even if unfriendly AI turns out to be no risk or in all possible worlds where it comes later. Both risks have the same utility x (if we don’t assume that an unfriendly AI could also wipe out aliens etc.). Thus .6x > .1x. But if the probability of solving friendly AI = A to the probability of solving the second risk = B is A ≤ (1/6)B then the expected utility of mitigating friendly AI is at best equal to the other existential risk because .6Ax ≤ .1Bx.
Consider that one order of magnitude more utility could easily be outweighed or trumped by an underestimation of the complexity of friendly AI.
So how hard is it to solve friendly AI?
Take for example Pascal’s mugging, if you can’t solve it then you need to implement a hack that is largely based on human intuition. Therefore, in order to estimate the possibility of solving friendly AI one needs to account for the difficulty in solving all sub-problems.
Consider that we don’t even know “how one would start to research the problem of getting a hypothetical AGI to recognize humans as distinguished beings.” [10]
P7 SIAI does not increase risks from AI.
By trying to solve friendly AI, SIAI has to think about a lot of issues related to AI in general and might have to solve problems that will make it easier to create artificial general intelligence.
It is far from being clear that SIAI is able to protect its findings against intrusion, betrayal, industrial or espionage.
P8 SIAI does not increase negative utility.
There are several possibilities by which SIAI could actually cause a direct increase in negative utility.
1) Friendly AI is incredible hard and complex. Complex systems can fail in complex ways. Agents that are an effect of evolution have complex values. To satisfy complex values you need to meet complex circumstances. Therefore any attempt at friendly AI, which is incredible complex, is likely to fail in unforeseeable ways. A half-baked, not quite friendly, AI might create a living hell for the rest of time, increasing negative utility dramatically [11].
2) Humans are not provably friendly. Given the power to shape the universe the SIAI might fail to act altruistic and deliberately implement an AI with selfish motives or horrible strategies [12].
P9 It makes sense to support SIAI at this time [13].
Therefore the probability of SIAI to be a worthwhile charity (P(CHARITY)) is the probability of the conjunction (P#∧P#) of its premises:
P(CHARITY) = P(P6∧P7∧P8∧P9)
As before, there are many more premises that need to be true in order for SIAI to be the best choice for someone who wants to maximize doing good by contributing money to a charity.
Further Reading
The following posts and resources elaborate on many of the above points and hint at a lot of additional problems.
Is an Intelligence Explosion a Disjunctive or Conjunctive Event?
Why an Intelligence Explosion might be a Low-Priority Global Risk
Notes and References
[1] Q&A with Shane Legg on risks from AI
[2] http://lukeprog.com/SaveTheWorld.html
[3] “In many ways, this is a book about hindsight. Pythagoras could not have imagined the uses to which his equation would be put (if, indeed, he ever came up with the equation himself in the first place). The same applies to almost all of the equations in this book. They were studied/discovered/developed by mathematicians and mathematical physicists who were investigating subjects that fascinated them deeply, not because they imagined that two hundred years later the work would lead to electric light bulbs or GPS or the internet, but rather because they were genuinely curious.”
17 Equations that changed the world
[4] Here is my list of “really stupid, frivolous academic pursuits” that have lead to major scientific breakthroughs.
Studying monkey social behaviors and eating habits lead to insights into HIV (Radiolab: Patient Zero)
Research into how algae move toward light paved the way for optogenetics: using light to control brain cells (Nature 2010 Method of the Year).
Black hole research gave us WiFi (ICRAR award)
Optometry informs architecture and saved lives on 9/11 (APA Monitor)
Certain groups HATE SETI, but SETI’s development of cloud-computing service SETI@HOME paved the way for citizen science and recent breakthroughs in protein folding (Popular Science)
Astronomers provide insights into medical imaging (TEDxBoston: Michell Borkin)
Basic physics experiments and the Fibonacci sequence help us understand plant growth and neuron development
http://blog.ketyov.com/2012/02/basic-science-is-about-creating.html
[5] “AIXI is often quoted as a proof of concept that it is possible for a simple algorithm to improve itself to such an extent that it could in principle reach superhuman intelligence. AIXI proves that there is a general theory of intelligence. But there is a minor problem, AIXI is as far from real world human-level general intelligence as an abstract notion of a Turing machine with an infinite tape is from a supercomputer with the computational capacity of the human brain. An abstract notion of intelligence doesn’t get you anywhere in terms of real-world general intelligence. Just as you won’t be able to upload yourself to a non-biological substrate because you showed that in some abstract sense you can simulate every physical process.”
Alexander Kruel, Why an Intelligence Explosion might be a Low-Priority Global Risk
[6] “…please bear in mind that the relation of Solomonoff induction and “Universal AI” to real-world general intelligence of any kind is also rather wildly speculative… This stuff is beautiful math, but does it really have anything to do with real-world intelligence? These theories have little to say about human intelligence, and they’re not directly useful as foundations for building AGI systems (though, admittedly, a handful of scientists are working on “scaling them down” to make them realistic; so far this only works for very simple toy problems, and it’s hard to see how to extend the approach broadly to yield anything near human-level AGI). And it’s not clear they will be applicable to future superintelligent minds either, as these minds may be best conceived using radically different concepts.”
Ben Goertzel, ‘Are Prediction and Reward Relevant to Superintelligences?’
[7] http://public.web.cern.ch/public/en/spotlight/SpotlightAandD-en.html
[8] “If any increase in intelligence is vastly outweighed by its computational cost and the expenditure of time needed to discover it then it might not be instrumental for a perfectly rational agent (such as an artificial general intelligence), as imagined by game theorists, to increase its intelligence as opposed to using its existing intelligence to pursue its terminal goals directly or to invest its given resources to acquire other means of self-improvement, e.g. more efficient sensors.”
Alexander Kruel, Why an Intelligence Explosion might be a Low-Priority Global Risk
[9] Section ‘Necessary resources for an intelligence explosion’, Why an Intelligence Explosion might be a Low-Priority Global Risk, Alexander Kruel
[10] http://lesswrong.com/lw/3aa/friendly_ai_research_and_taskification/
[12] http://lesswrong.com/lw/8c3/qa_with_new_executive_director_of_singularity/5y77
[13] “I think that if you’re aiming to develop knowledge that won’t be useful until very very far in the future, you’re probably wasting your time, if for no other reason than this: by the time your knowledge is relevant, someone will probably have developed a tool (such as a narrow AI) so much more efficient in generating this knowledge that it renders your work moot.”
Holden Karnofsky in a conversation with Jaan Tallinn
- How would you take over Rome? by (14 Mar 2012 16:24 UTC; 48 points)
- 's comment on Intelligence risk and distance to endgame by (13 Apr 2012 10:36 UTC; 4 points)
- 's comment on What is the best compact formalization of the argument for AI risk from fast takeoff? by (13 Mar 2012 18:35 UTC; 0 points)
- 's comment on Muehlhauser-Goertzel Dialogue, Part 1 by (17 Mar 2012 10:34 UTC; -14 points)
Imagine a group of 530 Spaniards...
At the risk of confirming every negative stereotype RationalWiki and the like have of us...have you read the Sequences? I’m reluctant to write a full response to this, but I think large parts of the Sequences were written to address some of these ideas.
I’m afraid I had the same reaction. XiXiDu’s post seems to take the “shotgun” approach of listing every thought that popped into XiXiDu’s head, without applying much of a filter. It’s exhausting to read. Or, as one person I know put it, “XiXiDu says a lot of random shit.”
I understand what you’re saying, but, speaking from a strictly nitpicky perspective, I don’t think the situation is analogous. The Roman Empire had many more soldiers to throw at the problem; much more territory to manage; comparatively better technology; and, perhaps more importantly, a much more robust and diverse—and therefore memetically resistant—society. They would therefore fare much better than the Aztecs did.
Conquistadors climbed to the top of a volcano to harvest sulphur for ammunition production. You can count on uploads in our society, as on some Navy Seals sent into the Roman world, to do analog actions. They both would not just wait for the help from nowhere. They would improvise as conquistadors once did.
Understood, but there’s only so much the conquistadors can do even with gunpowder. Guns can do a lot of damage against bronze swords and armor, but if they have more soldiers than you have bullets, then you’ll still lose.
Of course, if the conquistadors could build a modern tank, they’d be virtually invincible. But in order to do that, they’d need to smelt steel, vulcanize rubber, refine petroleum, manufacture electronics, etc. Even if they had perfect knowledge of these technologies, they couldn’t duplicate them in ye olde Aztec times, because such technologies require a large portion of the world’s population to be up to speed. There’s a limit to how much you can do armed with nothing but a pocket knife and a volcano.
I think this was XiXiDu’s point: knowledge alone is not enough, you also need to put in a lot of work (which is often measured in centuries) in order to apply it.
Understood that, too! But one can optimize and outsource a lot. Conquistadors employed Indians, enslaved Aztecs and Incas. Besides, the subjective time of an upload can be vast. A good idea can trim a lot of work need to be done. And at least my upload would be full of ideas.
Agreed; just as a single conquistador—or better yet, a modern engineer—transported into the Roman Empire would be full of ideas. He would know how to forge steel, refine petroleum, design electronic circuits, genetically engineer plants and animals, write software, plus many other things. But he wouldn’t be able to actually use most of that knowledge.
In order to write software, you need a computer. In order to build a computer, you need… well, you need a lot of stuff that outsourced Aztec (or Roman) slaves just wouldn’t be able to provide. You could enslave everyone on the continent, and you still wouldn’t be able to make a single CPU. Sure, if you were patient, very lucky, and long-lived, you could probably get something going within the next century or so. But that’s hardly a “FOOM”, and the Romans would have a hundred years to stop you, if they decided that your plans for the future aren’t to their liking.
Exactly. And here the parable breaks down. The upload just might have those centuries. Virtual subjective time of thousands of years to devise a cunning plan, before we the humans even discuss their advantage. Yudkowsky has wrote a short story about this. http://lesswrong.com/lw/qk/that_alien_message/
Bugmaster’s point was that it takes a century of action by external parties, not a century of subjective thinking time. The timetable doesn’t get advanced all that much by super-intelligence. Real-world changes happen on real-world timetables. And yes, the rate of change might be exponential, but exponential curves grow slowly at first.
And meanwhile, other things are happening in that century that might upset the plans and that cannot be arbitrarily controlled even by super-intelligence.
Err… minor quibble.
Exponential curves grow at the same rate all the time. That is, if you zoom in on the x^2 graph at any point at any scale, it will look exactly the same as it did before you zoomed in.
I think we are using “rate” in different ways. The absolute rate of change per unit time for an exponential is hardly constant; If you look at the segment of e^x near, say, e^10, it’s growing much faster than it is at e^(-10).
asr got my point exactly right.
Guns? I thought horses were their main advantage.
(What are the Aztecs gonna do, burn down all the grass in the continent?)
The OP used gunpowder as the example, so I went with it. You might be right about horses, though.
He’s read them well enough to collect a fairly complete index of cherry picked Eliezer quotes to try to make him look bad. I don’t think lack of exposure to prerequisite information is the problem here.
The index wedrifid was alluding to, if anyone cares: http://shityudkowskysays.tumblr.com/
I actually loved reading it. Some of those are up there among my favorite EY quotes. Arrogant, sometimes needing context to make them make sense and sometimes best left unsaid for practical reasons but still brilliant. For example:
There is also a quote there that I agree should remain visible, to Eliezer’s shame, until such time that he swallows his ego and publicly admits that it was an utterly idiotic way to behave. Then there is at least one quote which really deserves a disclaimer in a footnote—that EY has already written an entire sequence on admitting how stupid he was to think the way he thought when he wrote it!
I was actually rather disappointed when the list only went for a page or two. I was looking forward to reading all the highlights and lowlights. He deserves at least a few hundred best of and worst of quotes!
There’s always sorting in http://www.ibiblio.org/weidai/lesswrong_user.php?u=Eliezer_Yudkowsky
By following the link below the quote people could learn that he claims that he doesn’t agree with what he wrote there anymore. But I added an extra disclaimer now.
Thanks for making me find out what the Roko-thing was about :(
Yes, I created that blog. And it is not meant to give Eliezer Yudkowsky a bad name. The context of each quote is provided.
I think it is useful to emphasize some of the beliefs hold within this community.
Here is what someone over at Google+ wrote that I agree with,
Anyway, how could I possible libel someone by publishing what he and his followers believe to be true and good?
You know people are lazy; how many will click through to see the context? (Have you attached Javascript handlers to record click-throughs and compared them against the page traffic?)
How could I possibly libel someone by quoting out of context edited things he has written? “I did [...] have sex with that woman.”
What would you have done if you had meant to give him a bad name but nonetheless had to refrain from simply lying?
That’s fairly easy. There are many ways to do that, although he is already pretty good at it himself.
First I would start acting like Otto E. Rössler with respect to risks from AI. Then I would write as many AI researchers, computer scientists, popular bloggers and politicians etc. as possible about how THIS IS CRUNCH TIME, “it’s crunch time not just for us, it’s crunch time for the intergalactic civilization whose existence depends on us” And to back up my claims I would frequently cite posts and papers written by Yudkowsky and talk about how he is probably the most rational person alive and how most AI researchers are just biased.
No lies. Hands-free.
Not to nitpick or anything, but since you don’t actually seem to believe it’s “crunch time”, the strategy you outlined would indeed be a series of lies, regardless of whether Eliezer believes it true.
The tumblr strikes me as pretty much like the thing where you repeat what the other person says in a high-pitched voice, only minus the high-pitched voice.
Usual asshole talk from you again. I agree with 99.99% of what Eliezer writes. And just because he writes some shit doesn’t mean that I don’t respect him or that I want to give him a bad name.
Actually even you agree with me that he has been exaggerating some things in the past.
I agree with far less than that! Probably on the order of 95%. That’s higher than with most authors but I suspect lower than with, say, Yvain. Mind you Eliezer covers more controversial and original subjects than Yvain so I should expect Yvain to be more reliably correct than Eliezer. (Where from my perspective correctness cannot be distinguished from what I, all things including humility considered, agree with.)
Certainly. Enough so that ‘even [me]’ has less meaning when applied to myself than perhaps most people who have read all his posts multiple times. I’m also rather more free with publicly disagreeing with Eliezer (or anyone) than some others who may have specific disagreements with him. You often experience that as ‘asshole talk’. And quite rightly so. (Publicly) disagreeing with people is almost always instrumentally irrational.
I don’t want to be a jerk about this or belabor this point, but in order to decide exactly how I want to go in responding to this: have you read through all the sequences?
No, I can’t do that yet. It would take me half a year and I perceive other things to be more important, e.g. improving my math education.
Are you saying that people like me, who do not have the capabilities and time to read hundreds of posts, are excluded from asking about and discussing those issues? What do you suggest that I do, just ignore it? And where do you draw the line? Why is it sufficient to read the sequences? Why not exclude everyone who doesn’t understand Gödel machines and AIXI?
Well, we have to distinguish between social and epistemic spheres.
Socially it would be rude to exclude you from the debate or to say you don’t have a right to discuss these issues, especially when reading the Sequences is so time-consuming.
Epistemically, of course if you haven’t read the strongest arguments in favor of the importance of Friendly AI, you’re less likely to be right when you say it’s not important, and if gaining knowledge is time-consuming, then you either consume that time or continue lacking that knowledge.
Now there’s something tricky going on here. I can think of two extreme cases. The first is related to the courtier’s reply where some theist says an atheist can’t make atheist arguments unless she’s read every theologian from St. Augustine on and knows apologetics backwards and forwards; this is unfair and I think what you’re talking about. The second is where someone who knows nothing about medicine tries to argue with a neurosurgeon about his technique and the neurosurgeon tells him that he hasn’t been to medical school or read any of the literature on neurosurgery and so his opinions are completely meaningless; this one seems very fair and I would agree with the neurosurgeon.
I’m not really sure where the key difference lies, or which extreme this case falls into. But I asked if you’d read the Sequences for two reasons.
First, because I believed more or less what you believe now before I read the Sequences, and the Sequences changed my mind. If you’ve read the Sequences, then you might have found a flaw I missed and I should investigate what you’re saying further; if not, the simplest explanation is that you’re wrong for the same reasons I was wrong when I was in your position, and if you were to read the Sequences you’d have the same change of heart I did.
And second, because if you’d read the Sequences it would be worth debating some of these points with you; but since you haven’t looked at a much better piece that debates these points, I would recommend you do that instead, which would save me some time and make you more likely to be convinced. I realize that the Sequences are long, but that’s because supporting Eliezer’s view of Friendly AI is complicated and takes at least that long: I couldn’t write a reply to your thoughts which is as convincing as Eliezer’s in less space than it took Eliezer to write his. There’s a reason textbooks on neurosurgery aren’t light reading.
This is fascinating! Is there any way for you to expand on the following points:
What is it that you believed more or less before reading the Sequences?
What do you think is it that I believe with respect to risks from AI?
Is it possible to narrow down on what posts in particular made you change your mind?
I am not sure how much I have read. Maybe 30 posts? I haven’t found any flaws so far. But I feel that there are huge flaws.
How I feel is best exemplified by what Eliezer wrote about Pascal’s mugging, “I’d sooner question my grasp of “rationality” than give five dollars to a Pascal’s Mugger because I thought it was “rational”.”
Is there anything you could ask me, or make me do, that would enable you to find out if it is worth it for you to debate me, even if I haven’t read most of the Sequences?
P.S. Thank you for taking the time and effort.
Given the amount of activity you’ve applied to arguing about these topics (you wrote 82 LW posts during the last 1.5 years), I must say this is astonishing!
If you look closely, I haven’t written a single post and did not voice any criticism for years before the Roko incident.
I honestly didn’t expect there to be anything in the Sequences that could change my mind about the topic. Especially since smarter people than me read all of the Sequences and think that you are wrong.
It’s probably not entirely fair to compare my case to yours because I started reading the Sequences before I was part of this community, and so I was much less familiar with the idea of Friendly AI than you are. But to answer your questions:
Before reading the Sequences, I assumed unfriendly AI was one more crazy speculative idea about the future, around the level of “We’ll discover psionics and merge into a single cosmic consciousness” and not really worthy of any more consideration.
I think you believe that superintelligent AI may not be possible, that it’s unlikely to “go foom”, and that in general it’s not a great use of our time to worry about it.
That’s a good question. Looking over the post list I’m surprised that I can’t find any that look like the sort of thing that would do that directly (there’s a lot about how it’s important to build a Friendly AI as opposed to just throw one together and assume it will be Friendly, but if I understand you right we don’t disagree there). It could have been an indirect effect of realizing that the person who wrote these was very smart and he believed in it. It could have been that they taught me enough rationality to realize I might be wrong about this and should consider changing my mind. And it could have been just very gradual worldview change. You said you were reading the debate with Robin, and that seems like a good starting point. The two dependency thingies labelled “Five Sources of Discontinuity” and “Optimization and the Singularity” here also give me vague memories of being good. But I guess that either I was wrong about the Sequences being full of brilliant pro-Singularity arguments, or they’re more complicated than I thought. Maybe someone else who’s read them more recently than I have can answer this better?
...which shouldn’t discourage you from reading the Sequences. They’re really good. Really. They might or might not directly help you on this question, but they’ll be indirectly helpful on this and many other things. It’s a really good use of your time (debating with me isn’t; I don’t claim any special insight on this issue beyond what I’ve picked up from the Sequences and elsewhere, and I don’t think I’ve ever posted any articles on AI simply because I wouldn’t even meet this community’s lax standards for expertise).
(Addendum to my other comment)
Here is why I believe that reading the Sequences might not be worth the effort:
1) According to your survey, 38.5% of all people have read at least 75% of the Sequences yet only 16.5% think that unfriendly AI is the most fearsome existential risk.
2) The following (smart) people have read the Sequences, and more, but do not agree about risks from AI:
Robin Hanson
Katja Grace (who has been a visiting fellow)
John Baez (who interviews Eliezer Yudkowsky)
Holden Karnofsky
Ben Goertzel
So what? I’m not even sure that Eliezer himself considers uFAI the most likely source of extinction. It’s just that Friendly AI would help save us from most the other possible sources of extinction too (not just from uFAI), and from several other sources of suffering too (not just extinction), so it kills multiple birds with one stone to figure it out.
As a point of note, I myself didn’t place uFAI as the most likely existential risk in that survey. That doesn’t mean I share your attitude.
I hope I didn’t claim the Sequences, or any argument were 100% effective in changing the mind of every single person who read them.
Also, Ben Goertzel has read all the Sequences? That makes that recent conversation with Luke kind of sad.
No, but in the light of an expected utility calculation. Why would I read the Sequences?
Assuming you continue to write posts authoritatively about subjects related to said sequences—including criticisms of the contents therein—having read the sequences may reduce the frequency of you humiliating yourself.
They contain many insights unrelated to AI (looking at the sequences wiki page, it seems that most AI-ish things are concentrated in the second half). And many people had fun reading them. I think it would be a better use of time than trying to generically improve your math education that you speak of elsewhere (I don’t think it makes sense to learn math as an instrumental goal without a specific application in mind—unless you simply like math, in which case knock yourself out).
From a theoretical standpoint, you should never expect that observing something will shift your beliefs in some particular direction (and, guess what, there’s a post about that). This doesn’t work for humans—we can be convinced of things and we can expect to be convinced even if we don’t want to. But then, the fact that the sequences fail to convince many people shouldn’t be an argument against reading them. At least now you can be sure that they’re safe to read and won’t brainwash you.
I do not believe that it is that speculative and I am really happy that there are people like Eliezer Yudkowsky who think about it.
In most of my submissions I try to show that there are a lot of possibilities of how the idea of superhuman AI could fail to be a risk.
Why do I do that? The reason for doing so are due to the main points of disagreement with Eliezer Yudkowsky and others who believe that “this is crunch time”. The points being that 1) I believe that they are overconfident when it comes to risks from AI, that the evidence simply doesn’t allow you to dramatize the case the way they do, and 2) I believe that they are overconfident when it comes to their methods of reasoning.
I would never have critized them if they had said, 1) “AI might pose a risk. We should think about it and evaluate the risk carefully.” and 2) “Here are some logical implications of AI being a risk. We don’t know if AI is a risk so those implications are secondary and should be discounted accordingly.”
But that is not what is happening. They portray friendly AI as a moral imperative and use the full weight of all logical implications of risks from AI to blow up its expected utility.
And that’s where my saying that I “found no flaws but feel that there are flaws” comes into play.
I understand that P(Y|X) ≈ 1, then P(X∧Y) ≈ P(X). The problem is that, as muflax put it, I don’t see how you can believe in the implied invisible and remain even remotely sane. It does not work out. Even though on an intellectual level I completely agree with it, my intuition is yelling that something is very wrong here. It is ear-battering. I can’t ignore it. Call it irrational or just sad, I can’t help it.
It is fascinating. If I could work directly at it then I would do it. But giving away my money? Here we get to point #1, mentioned above.
Is there enough evidence that my money would make a difference? This question is deep. The question is not just about the likelihood of a negative Singularity, but also the expected utility of contributing any amount of money to friendly AI research. I am seriously unable to calculate that. I don’t even know if I should get an MRI to check for unruptured brain aneurysms.
Another problem is that I am not really altruistic. I’d love to see everybody happy. But that’s it. But then I also don’t really care about myself that much. I only care if I might suffer, but not about being dead. That’s what makes the cryonics question pretty easy for me. I just don’t care enough.
This is one of the things I don’t understand. I don’t think Eliezer is that smart. But even if he was, I don’t think that increases the probability of him being right about some extraordinary ideas very much. Especially since I have chatted with other people that are equally smart who told me that he is wrong.
There are many incredible smart people who hold really absurd ideas.
The biggest problem is that he hasn’t achieved much. All he did was putting some of the work of other people together, especially in the field of rationality and heuristics and biases. And he wrote a popular fanfic. That’s it.
Yeah, he got some rich people to give him money. But the same people also support other crazy ideas with the same amount of money. That’s little evidence.
Sure, I am very likely wrong. But that argument cuts both ways.
I will try. Right now I am very off-put by Eliezer’s style of writing. I have a hard time to understand what he is saying while Robin is very clear and I agree about like everything he says.
But I will try to continue and research everything I don’t understand.
In what respect? Those posts that I have read were quite interesting. But I even enjoy reading a calculus book right now. And just as I expect to never actually benefit from learning calculus I don’t think that it is instrumentally useful to read the Sequences. It is not like that I am raving mad. I have enough rationality to live a good life without the Sequences.
If you mean that they are good in convincing you of risks from AI, then I also ask you how sure you are that they are not only convincing but actually factually right? Do you believe that you have the expertise that is necessary to discern a good argument about artificial intelligence from one that is not even wrong?
Just one last question if you allow. What are you doing against risks from AI? Do you pursue a carrier where you can earn a lot of money to contribute it to SIAI?
What’s your numerical probability estimate that, assuming no one puts much work into stopping it, Unfriendly AI will seriously damage or destroy human civilization in the next few centuries?
Mine is...hmm...I don’t know. Maybe 50%? I’m not sure. I do know that if there were an asteroid nearby with the same probability of impacting Earth, I’d be running up to people and shaking them and shouting “WHY AREN’T WE BUILDING MORE ASTEROID DEFLECTORS?! WHAT’S WRONG WITH YOU? PEOPLE!” I don’t know if I believe in unconditional moral imperatives, but if there were a 50% chance of an asteroid striking Earth soon, or even a 10% chance, and no one was doing anything about it, I would at least feel an imperative conditional on some of my other beliefs to try to help stop it.
So maybe part of what the Sequences did for me was help calibrate my brain well enough so that I noticed the similarity between the asteroid and the AI case.
The remaining sane part seems to be a matter of personal psychology. My undergrad philosophy prof once told a story of how a classmate of hers had to be committed to an institution after reading a book on nihilism: he just started doubting everything and went mad. My prof read the same book on nihilism, thought it made some interesting arguments that she couldn’t immediately refute, went back to her everyday life, and a few years later reconsidered it and found some possible refutations.
I have always sympathized more with my professor’s point of view: I can read arguments which if taken seriously would be nightmarish or imply total doubt, admit the arguments seem plausible, and then go back to making pizza or doing homework or whatever. I’m not sure what the difference is between people like myself and my professor, and people like you and my professor’s classmate. Maybe you’re more rational, deep down? Or maybe you’re just naturally depressed and anxious, and your brain latches onto this as an excuse? I don’t know. In any case, I don’t at all think of being able to resist terrifying implications as a rationalist skill and I’m not sure what I would do in your position.
(in my own case, I have pretty much decided to live a normal life but give a reasonable amount of what I make to SIAI and associated charities, probably, volunteer for them if they need it, and leave it at that. Are the algorithms that produced this plan optimal? No. Are they a heck of a lot better than going insane and resisting the idea of friendly AI with all my might because if I accepted it I would have to go insane and give away all my money? Yes.)
...I would describe this as being altruistic; also, I share your intuitions about death and cryonics.
Okay, point. I guess I got the impression Eliezer was both smart and rational; that he was smart in exactly the way that ought to prevent him from holding absurd ideas. This is an intuition, so I can’t really justify it.
This seems like a fully general counterargument. “Sure, the evidence for evolution sounds convincing; but how do you know it’s actually true and you aren’t just being tricked?”
I’m pursuing a career as a doctor. Despite a recent major setback, I’m still hoping to get there within a year or so. After that, yeah, I do intend to donate a lot to SIAI—albeit, as I said before, I don’t claim I’ll be anywhere near perfect.
That would be more than enough to devote a big chunk of the world’s resources on friendly AI research, given the associated utility. But you can’t just make up completely unfounded conjectures, then claim that we don’t have evidence either way (50% chance) but that the utility associated with a negative outcome is huge and we should therefore take it seriously. Because that reasoning will ultimately make you privilege random high-utility outcomes over theories based on empirical evidence.
You can’t really compare that. The arguments for evolution are pretty easy to understand and the evidence is overwhelming. But Eliezer Yudkowsky could tell me anything about AI and I would have no way to tell if he was right or not even wrong.
I see. That makes me take you much more seriously.
You know, one upside of logic is that, if someone tells you proposition x is true, gives you the data, and shows their steps of reasoning, you can tell whether they’re lying or not. I’m not a hundred percent onboard with Yudkowsky’s AI risk views, but I can at least tell that his line of reasoning is correct as far as it goes. He may be making some unjustified assumptions about AI architecture, but he’s not wrong about there being a threat. If he’s making a mistake of logic, it’s not one I can find. A big, big chunk of mindspace is hostile-by-default.
I’m not rich. My gross annual salary is lower than Eliezer Yudkowsky’s or Nick Bostrom’s. (mentioned since you keep using me as your example commenter for donations)
I just spent over a minute trying to emphasize the ‘argument from blatant ignorance’ point here in a way that I didn’t just seem petty. I don’t think I succeeded.
In my professional life, I am frequently asked questions about a complex system I have some responsibility for. For pieces of that system, I got sufficiently tired of answering certain recurring questions that I wrote a set of documents that addressed those areas.
How would you recommend I reply to someone who asks me questions about those areas and, when directed to the documents I wrote, replies that they don’t have time to read the document, they just want to participate in a conversation with me about the subject in which I answer their specific questions?
I usually deal with these types of questions by sending them a link to a specific document. For example, I might say, “take a look at the flow diagram at ${url}, then read the notes on ${node}, they will explain why you are getting that error”. If the person comes back and says, “I read your notes but I have no idea what they mean”, I’d point him to some introductory material; but in practice, this happens rarely, because my notes are full of links.
One thing I used to do, but stopped doing, was to say, “open up the project wiki and read through all the pages in it”. The people who come to me with questions are looking for an effective solution to a specific problem, not for general education and/or enlightenment. They have tangible goals just like I do, after all, and there are only so many hours in a day.
You could answer in the same way that a lot of AI researchers and computer scientists did, who I wrote (400+ by now): “Your fears are the result of a lack of knowledge. Come back once you have enough background to ask intelligent questions about AI.”
But then nobody learns anything because everyone thinks that they are obviously right and the others are idiots.
You might have seen the following sentences in some of my recent submissions:
″...highly specific, conjunctive, non-evidence-backed speculations on possible bad outcomes.”
″… in my view they’ve just sprinkled enough mathematics and logic over their fantasies to give them a veneer of respectability.”
″… I … personally think that they are naïve as far as the nature of human intelligence goes. I think they are mostly very bright and starry-eyed adults who never quite grew out of their science-fiction addiction as adolescents. None of them seems to have a realistic picture about the nature of thinking...”
Those lines are copied straight out of emails from people who know a lot more than I do.
So what do I recommend? I recommend that both sides do the same that evolutionary biologist did when creationists attacked. Write a book on it or create documents that can actually be read by people with a large inferential distance. But don’t tell them to read hundreds of blog posts that are largely unrelated to the problem in question, or your papers that only someone with a doctorate in machine learning can understand.
OK.
So, suppose I decide to write a book on it, as you suggest.
What do you recommend I do during the years that that book-writing process is going on?
And does that recommendation change if it turns out that I’m wrong and merely think that I’m right, vs. if I’m actually right?
Talk about the subject of the book as much as possible, even if you talk bullshit. Because the book was meant to explain why you are talking bullshit. People talking bullshit is exactly what it takes to write a good book on dissolving bullshit.
No. See ‘Darwin’s Dangerous Idea’ by Daniel Dennett. The book starts by reviewing all the bullshit people have been saying in the past few hundred years and manages to shed light on culture, popular misconceptions and how you can be wrong.
He actually mentions a few times how creationists and other enemies of evolution actually allowed evolutionary biologists to hone their arguments and become stronger. And yes, the arguments of the critics were often poor from the point of view of the experts, but strong from the point of view of laymen.
OK, thanks… that’s clear.
For what it’s worth, I disagree with your first recommendation.
I do agree that Dennett (both in DDI and more generally) has an admirable willingness to engage with the “bullshit” in his field, though he also is willing to unilaterally dismiss vast swaths of it when he decides it’s no longer valuable for him to engage with (see, for example, his treatment of qualia in ‘Consciousness Explained’… or listen to him talk to undergraduates, if he still does that; he was always a treat to listen to).
No. He’s saying:
However, I will say that EY wrote at least part of the sequences because he got sick and tired of seeing people who try to reason about AI fall immediately into some obvious failure state. You have a tendency to fall into these failure states; e.g., Generalizing from Fictional Evidence during the whole Rome Sweet Rome allusion.
Would you rather continue running around in circles, banging your head against the wall? Even if you did read the sequences, there’d still be no guarantee that you wouldn’t continue doing the same thing. But, to paraphrase Yudkowsky, at least you’d get a saving throw.
Nobody said anything of the sort. Again, Yvain’s trying to formulate a response.
I had to say this like 10 times now. I am getting the impression that nobody actually read what I wrote.
The whole point was to get people thinking about how an AI is actually going to take over the world, in practice, rather than just claiming it will use magic.
If the IRC channel and the discussion section is any evidence, all you’ve managed to accomplish is to get people to think about how to take over ancient Rome using modern tech.
No, you don’t, as this post alone demonstrates. This is why I consider you a troll.
That wouldn’t work out well...
That you got 22 upvotes for that is incredible sad.
So wiping out the Aztecs isn’t enough, you wanted the Spaniards to quickly establish full control over the Mayan homelands as well?
That seems to me like moving the goalposts, you no longer just want the scientists to wipe out the Roman Empire, you now also want them to “establish full control” over both the Romans’ former territory and the neighboring Persian empire?
Downvoted for the continuing rudeness, and the continuing whining about how people vote others.
Also downvoted for the aforementioned “moving the goalposts”.
I never argued that a hundred people from today, given machine guns, would have no chance to wipe out the roman Senate and kill the the roman consul. Which doesn’t mean that an AI could do the same. But even if it was to wipe out Washington, that would not constitute world domination.
The Spaniards, with the the help of their Tlaxcallan allies, managed to kill the ruler and destabilize Aztec empire and their domination. Yet most of the Mesoamerican cultures were intact afterwards. It took over a hundred years to defeat those.
Anyway, that incident can’t be compared with the Roman empire and much less with the world of today.
You seem to have no clue what my goal was in the first place. I tried to exemplify the difficulties of taking over the world by an example that bears some similarity to the idea that an AI might just take over the world. In reality it would be much more difficult for an AI to do the same due to its fragility and its technological and civilizatory dependencies.
I am not whining about voting behavior but that people who apparently try to be rational could be fooled that easily and act like a cult to protect their beliefs against the least amount of criticism.
You are wrong and any amount of downvoting won’t make you right.
The question was specific: “Could such a group easily wipe away the Roman empire when beamed back in time?”
Ofcourse you can define “wipe away” as by necessity something even more extreme than what the Goths or the Ottomans managed to do with the actual Roman Empires.
On this thread alone you’ve complained about the votes on three different comments, Yvain’s, asr’s, and gwern’s. Your actions as observed from outside are indistinguishable from those of someone who is whining about voting behaviour.
You behaved much more aggressively and much more rudely at (even less) criticism towards this post of yours, than I’ve seen leveled against you.
I’m interested in protecting the standards of polite discourse that are part of LessWrong, which said standards of polite discourse you seem determined to destroy by continuous and gratituous insults. I haven’t even touched any of the actual content of your post, only commented on your rude responses to criticism of it.
I agree that it’s sad I got 22 upvotes; it was a cheap shot at an example I don’t think was really relevant. But here’s another cheap shot: if you learned a future history book would say “With the help of tens of trillions of nanobots, it would take more than 170 years for unfriendly superintelligences to eradicate the last pockets of human resistance beneath the crust of Ganymede”, would you follow that up with “See, I was right, it wasn’t such a big deal after all”?
I agree with all Aris Katsaris’ points as well, but, again, I don’t think this example is too relevant to AI. Although this whole concept of historical analogies for AI isn’t great, a slightly better example than “530 Spaniards vs. Mesoamerica”, might be “1 Spaniard infected with smallpox vs. Mesoamerica”. AIs don’t have to play fair.
Full stop. If advanced nanotechnology comes first then my above post is rendered obsolete. I do completely agree that an AI in possession of advanced nanotech assemblers will be able to take over the world.
So, the AI, even if coming before advanced nanotechnology, would just need to invent some advanced nanotechnology and convince some people to manufacture them, is that right?
I’m pretty sure UFAI isn’t even necessary for advanced nanotech to take over the world.
All your counter-arguments are enthymematic; as far as I can tell, you are actually arguing against a proposition which looks more like
I would find your enthymematic far more convincing if you explained why things like Goedel machines are either fallacious or irrelevant.
Your argument is basically an argument from fiction; it’s funny that you chose that example of the Roman Empire when recently Reddit spawned a novel arguing that a Marine Corps (surely less dangerous than your 100) could do just that. I will note in passing that black powder’s formulation is so simple and famous that even I, who prefers archery, knows it: saltpeter, charcoal, and sulfur. I know for certain that the latter two are available in the Roman empire and suspect the former would not be hard to get. EDIT: and this same day, a Mafia-related paper I was reading for entertainment mentioned that Sicily—one of the oldest Roman possessions—was one of the largest global exporters of sulfur in the 18th/19th centuries. So that ingredient is covered, in spades!
A civilization which exists and is there for the taking.
Chimp brains have not improved at all, even to the point of building computers. There is an obvious disanalogy here...
All of which are available to a ‘simple algorithm’. Artificial life was first explored by von Neumann himself!
Are you serious? Are you seriously claiming this? Dead-simple chess and Go algorithms routinely turn out fascinating moves. Genetic algorithms are renowned for producing results which are bizarre and inhuman and creative. Have you never read about the famous circuit which has disconnected parts but won’t function without them?
What is this bullshit ‘computers can’t exhibit creativity’ doing here? Searle, why did you steal XiXiDu’s account and post this?
‘I may be completely wrong, but hey, I can still ask rhetorically whether I’m not actually right!’
This implies P2.
Why can’t I predict the next move of my chess algorithm? Why is there no algorithm to predict the AI algorithm simpler and faster than the original AI algorithm?
This is just naive. Source code can be available and either the maliciousness not obvious (see the Underhanded C Contest) or not prove what you think it proves (see Reflections on Trusting Trust, just for starters). Assuming you are even inspecting all the existing code rather than a stub left behind to look like an AI.
No. Not all the premises are necessary, so a conjunction is inappropriate and establishes a lower bound, at best.
I’m going to stop here. This might have been a useful exercise if you were trying to establish solely necessary premises, in the same vein as Chalmer’s paper or Drake equation-style examination of cryonics, but you’re not doing that.
I disagree with the gist of your comment, but I upvoted it because this quote made me LOL.
That said, I don’t think that XiXiDu is claiming that computers can’t exhibit creativity, period. Rather, he’s saying that the kind of computers that SIAI is envisioning can’t exhibit creativity, because they are implicitly (and inadvertently) designed not to.
You are arguing past each-other. XiXiDu is saying that a programmer can create software that can be inspected reliably. We are very close to having provably-correct kernels and compilers, which would make it practical to build reliably sandboxed software, such that we can look inside the sandbox and see that the software data structures are what they ought to be.
It is separately true that not all software can be reliably understood by static inspection, which is all that the underhanded C contest is demonstrating. I would stipulate that the same is true at run-time. But that’s not the case here. Presumably developers of a large complicated AI will design it to be easy to debug—I don’t think they have much chance of a working program otherwise.
No, you are ignoring Xi’s context. The claim is not about what a programmer on the team might do, it is about what the AI might write. Notice that the section starts ‘The goals of an AI will be under scrutiny at any time...’
Yes. I thought Xi’s claim was that if you have an AI and put it to work writing software, the programmers supervising the AI can look at the internal “motivations”, “goals”, and “planning” data structures and see what the AI is really doing. Obfuscation is beside the point.
I agree with you and XiXiDu that such observation should be possible in principle, but I also sort of agree with the detractors. You say,
Oh, I’m sure they’d try. But have you ever seen a large software project ? There’s usually mountains and mountains of code that runs in parallel on multiple nodes all over the place. Pieces of it are usually written with good intentions in mind; other pieces are written in a caffeine-fueled fog two days before the deadline, and peppered with years-old comments to the extent of, “TODO: fix this when I have more time”. When the code breaks in some significant way, it’s usually easier to write it from scratch than to debug the fault.
And that’s just enterprise software, which is orders of magnitude less complex than an AGI would be. So yes, it should be possible to write transparent and easily debuggable code in theory, but in practice, I predict that people would write code the usual way, instead.
You are just lying. Some of what I wrote:
What asr wrote was just much more clearly.
It is incredible sad that your comment is at 0 and a bunch of fallacious accusations by gwern are at +20.
I hate fallacious arguments by gwern at least as much as the next guy but in this particular instance they were straightforward enough.
I happily admit when I see a straightforward argument. As for example his argument about double-counting probabilities. I have been simply wrong there. But the rest of the comment was not even close to constituting a good argument against anything I wrote in the OP and some of it were just straw men.
Yes, a thorough analysis would take a long time and I am not the right person to do that. I only have the capability to improve it incrementally.
The reason for why I post something like this anyway is that SIAI and people like you are exclusively stating that which speaks in favor of your worldview, without any critical analysis of your beliefs. And if someone else does it for you then you make accusations in the most hypocritical manner possible.
I can’t review the work of Jürgen Schmidhuber because I lack the mathematical background. But you knew that.
If his work was relevant in estimating risks from AI then it is up to people like you to write about it and show how his work does constitute evidence for your claims.
I did the best I can do. I even interviewed a bunch of AI researchers and asked others about his work in private.
Have you taken the effort to ask actual experts? You people mainly rely on surveys that ask for time-frames and then interpret that to mean that risks from AI are nigh. Even though most AI researchers who answered that AGI will happen soon would deny the implications that you assume. Which is just another proof of your general dishonesty and conformation bias.
So? What’s your point?
This wouldn’t be nearly enough.
Magical thinking.
Yes.
Yes, what I said.
I am aware of that. Not sure what’s your point though.
I never argued that.
The point was that humans can use the same technique that the AI does. I never claimed that it would be possible to predict the next move.
This is just naive. We’re talking about a plan for world domination, which doesn’t just include massive amounts of code that would have to be hidden from inspection but also massive amount of actions.
Come on. Maybe you disagree with gwern’s response and think he missed a bunch of your points, but this is just name-calling. I like your posts, but a comment like this make me lose respect for you.
I know nothing about you.
His papers and those of Legg or Hutter are all online. I’ve hosted some of them myself eg. the recent Journal of Consciousness Studies one. The abstracts are pretty clear. They’ve been mentioned and discussed constantly on LW. You yourself have posted material on them, and material designed as an introduction for relative beginners so hopefully you read & learned from it.
So unless they agree in every detail, their forecasts are useless?
That story is an intuition pump (not one of my favorites, incidentally) - and your story is a pump with a broken-off handle.
Gee, I don’t suppose you would care to enlarge on that.
Which means what, exactly? It’s magical thinking to point out that our current civilization exists and is available to any AI we might make?
You said they necessarily lack creativity.
I’ll reiterate the quote:
Next:
So in other words, we would be able to detect the AI had gone bad while it was in the process of executing the massive amount of actions of taking over the world. I agree! Unfortunately, that’s may not be a useful time to detect it...
I can’t speak for XiXiDu, but I myself have noticed a bit of magical thinking that is sometimes employed by proponents of AGI/FAI. It goes something like this (exaggerated for effect):
1). It’s possible to create an AI that would recursively make itself smarter
2). Therefore the AI would make itself very nearly infinitely smart
3). The AI would then use its intelligence to acquire godlike powers
As I see it, though, #2 does not necessarily follow from #1, unless one makes an implicit assumption that Moore’s Law (or something like it) is a universal and unstoppable law of nature (like the speed of light or something). And #3 does not follow from #2, for reasons that XiXiDu articulated—even if we assume that godlike powers can exist at all, which I personally doubt.
If you took the ten smartest scientists alive in the world today, and transported them to Ancient Rome, they wouldn’t be able to build an iPhone from scratch no matter how smart they were. In addition, assuming that what we know of science today is more or less correct, we could predict with a high degree of certainty that no future scientist, no matter how superhumanly smart, would be able to build a perpetual motion device.
Edited to add: I was in the process of outlining a discussion post on this very subject, but then XiXiDu scooped me. Bah, I say !
I’d still like to see you write it, if it’s concise.
#2 does not need to follow since we already know it’s false—infinite intelligence is not on offer by the basic laws of physics aside from Tipler’s dubious theories. If it is replaced by ‘will make itself much smarter than us’, that is enough. (Have you read Chalmer’s paper?)
Which reasons would those be? And as I’ve pointed out, the only way to cure your doubt if the prior history of humanity is not enough would be to actually demonstrate the powers, with the obvious issue that is.
Ok, but how much smarter ? Stephen Hawking is much smarter than me, for example, but I’m not worried about his existence, and in fact see it as a very good thing, though I’m not expecting him to invent “gray goo” anytime soon (or, in fact, ever).
I realize that quantifying intelligence is a tricky proposition, so let me put it this way: can you list some feats of intelligence, currently inaccessible to us, which you would expect a dangerously smart AI to be able to achieve ? And, segueing into #3, how do these feats of intelligence translate into operational capabilities ?
Probably not; which paper are you referring to ?
The ones I alluded to in my next paragraph:
The problem here is that raw intelligence is not enough to achieve a tangible effect on the world. If your goal is to develop and deploy a specific technology, such as an iPhone, you need the infrastructure that would supply your raw materials and labor. This means that your technology can’t be too far ahead of what everyone else in the world is already using.
Even if you were ten times smarter than any human, you still wouldn’t be able to conjure a modern CPU (such as the one used in iPhones) out of thin air. You’d need (among other things) a factory, and a power supply to run it, and mines to extract the raw ores, and refineries to produce plastics, and the people to run them full-time, and the infrastructure to feed those people, and a government (or some other hegemony) to organize them, and so on and so forth… None of which existed in Ancient Rome (with the possible exception of the hegemony, and even that’s a stretch). Sure, you could build all of that stuff from scratch, but then you wouldn’t be going “FOOM”, you’d be going “are we there yet” for a century or so (optimistically speaking).
Are you referring to some specific historical events ? If so, which ones ?
Okay, you are right. I was wrong to expect you to have read my comments where I explained how I lack the most basic education (currently trying to change that).
Yeah, I post a lot of stuff that I sense to be important and that I would love to be able to read. I hope to be able to do so in future.
No. But if you are interested in risks from AI you should ask them about risks from AI and not just about when human-level AI will likely be invented.
My story isn’t a story but a quickly written discussion post, as a reply to post where an argument in favor of risks from AI has been outlined that was much too vague to be useful.
The problem I have is that it can be very misleading to just state that it is likely physically possible to invent smarter than human intelligence that could then be applied to its own improvement. It misses a lot of details.
Show me how that is going to work out. Or at least outline how a smarter-than-human AI is supposed to take over the world. Why is nobody doing that?
Just saying that there will be “a positive feedback loop in which an intelligence is making itself smarter” makes it look like something that couldn’t possible fail.
100 people are not enough to produce and employ any toxic gas or bombs in a way that would defeat a wide-stretched empire with many thousands of people.
It is magically thinking because you don’t know how that could possible work out in practice.
I said that there is nothing but evolution, a simple algorithm, when it comes to creativity and the discovery of unknown unknowns. I said that the full potential of evolution can only be tapped by a society of minds and its culture. I said that it is highly speculative that there exists a simple algorithm that would constitute a consequentialist AI with simple values that could achieve the same as aforementioned society of minds and therefore work better than evolution.
You just turned that into “XiXiDu believes that simple algorithms can’t exhibit creativity.”
Well, then we agree.
People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything. The scenarios will never be enough for critics because until they are actually executed there will always be some doubt that they would work—at which point there would be no need to discuss them any more. Just like in cryonics (if you already had the technology to revive someone, there would be no need to discuss whether it would work). This is intrinsic to any discussion of threats that have not already struck or technologies which don’t already exist.
I am reminded of the quote, “‘Should we trust models or observations?’ In reply we note that if we had observations of the future, we obviously would trust them more than models, but unfortunately observations of the future are not available at this time.”
Because that’s the best way to take over...
That is not what you said. I’ll requote it:
If a singleton lacks feedback from diversity and something which is the ‘cornerstone’ of diversity is something a singleton cannot have… This is actually even stronger a claim than simple algorithms, because a singleton could be a very complex algorithm. (You see how charitable I’m being towards your claims? Yet no one appreciates it.)
And that’s not even getting into your claim about spectrum of research, which seems to impute stupidity to even ultraintelligent agents.
(‘Let’s see, I’m too dumb to see that I am systematically underinvesting in research despite the high returns when I do investigate something other than X, and apparently I’m also too dumb to notice that I am underperforming compared to those oh-so-diverse humans’ research programs. Gosh, no wonder I’m failing! I wonder why I am so stupid like this, I can’t seem to find any proofs of it.’)
Speaking as one of the critics, I’ve got to say that these scenarios are “not enough” for me not because there’s “some doubt that they would work”, but because there’s massive doubt that they would work. To use an analogy, I look both ways before crossing the street because I’m afraid of being hit by a car; but I don’t look up all the time, despite the fact that a meteorite could, theoretically, drop out of the sky and squash me flat. Cars are likely; meteorites are not.
I could elaborate regarding the reasons why I doubt some of these world takeover scenarios (including “hacking the Internet” and “eating everything”), if you’re interested.
Not really. Any scenario presented in any level of detail can be faulted with elaborate scenarios why it would not work. I’ve seen this at work with cryonics: no matter how detailed a future scenario is presented or how many options are presented in a disjunctive argument, no matter how many humans recovered from death or how many organs preserved and brought back, there are people who just never seem to think it has a non-zero chance of working because it has not yet worked.
For example, if I wanted to elaborate on the hacking the Internet scenario, I could ask you your probability on the possibility and then present information on Warhol worm simulations, prevalence of existing worms, number of root vulnerabilities a year, vulnerabilities exposed by static analysis tools like Coverity, the early results from fuzz testers, the size of the computer crime blackmarket, etc. until I was blue in the face, check whether you had changed your opinion and even if you said you changed it a little, you still would not do a single thing in your life differently.
Because, after all, disagreements are not about information. There’s a lot of evidence reasoning is only about arguing and disproving other people’s theories, and it’s increasingly clear to me that politics and theism are strongly heritable or determined by underlying cognitive properties like performance on the CRT or personality traits; why would cryonics or AI be any different?
The point of writing is to assemble useful information for those receptive, and use those not receptive to clean up errors or omissions. If someone reads my modafinil or nicotine essays and is a puritan with regard to supplements, I don’t expect them to change their minds; at most, I hope they’ll have a good citation for a negative point or mention a broken hyperlink.
That is a rather uncharitable interpretation of my words. I am fully willing to grant that your scenarios are possible, but are they likely ? If you showed me a highly detailed plan for building a new kind of skyscraper out of steel and concrete, I might try and poke some holes in it, but I’d agree that it would probably work. On the other hand, if you showed me a highly detailed plan for building a space elevator out of candy-canes, I would conclude that it would probably fail to work. I would conclude this not merely because I’ve never seen a space elevator before, but also because I know that candy-canes make a poor construction material. Sure, you could postulate super-strong diamondoid candy-canes of some sort, but then you’d need to explain where you’re going to get them from.
For the record, I believe that cryonics has a non-zero chance of working.
I think this would depend on how much my opinion had, in fact, changed. If you’re going to simply go ahead and assume that I’m a disingenuous liar, then sure, there’s no point in talking to me. Is there anything I can say or do (short of agreeing with you unconditionally) to prove my sincerity, or is the mere fact of my disagreement with you evidence enough of my dishonesty and/or stupidity ?
And yet, de-converted atheists as well as converted theists do exist. Perhaps more importantly, the above sentence makes you sound as though you’d made up your mind on the topic, and thus nothing and no one could persuade you to change it in any way—which is kind of like what you’re accusing me of doing.
But none of them make any sense to me, see below.
Wait, your quote said what I said I said you said I didn’t say.
I have no idea. You don’t have any idea either or you’d have told me by now. You are just saying that magic will happen and the world will be ours. That’s the problem with risks from AI.
See, that’s the problem. The AI can’t acquire the resources that are necessary to acquire resources in the first place. It might figure out that it will need to pursue various strategies or build nanoassemblers, but how does it do that?
Taking over the Internet is no answer, because the question is how. Building nanoassemblers is no answer, because the question is how.
We have plenty of ideas. Yvain posted a Discussion thread filled with ideas how. “Alternate history” is an old sub-genre dating back at least to Mark Twain (who makes many concrete suggestions about how his Connecticut yankee would do something similar).
But what’s the point? See my reply to Bugmaster—it’s impossible or would defeat the point of the discussion to actually execute the strategies, and anything short of execution is vulnerable to ‘that’s magic!11!!1’
By reading the many discussions of what could go wrong and implementing whatever is easiest, like hacking computers. Oh the irony!
Your comment is too long to reply today. Just one quick remark:
The point is that P2 does not imply P3, yet P2 has to be true in the first place.
By covering premises which are subsumed or implied by other premises, you are engaged in one of the more effective ways to bias a conjunctive or necessary analysis: you are increasing the number of premises and double-counting probabilities. By the conjunction rule, this usually decreases the final probability.
I’ve pointed out before that use of the conjunction approach can yield arbitrarily small probabilities based on how many conjuncts one wishes to include.
For example, if I were to argue that based on methodological considerations one could never have greater than 99% confidence in a theory and say that none of the premises could therefore be more than 0.99, I can take a theory of 2 conjuncts with 0.99^2 =98% maximum confidence and knock it down to 94% solely by splitting each conjunct into 3 premises (‘this premise conceals a great deal of complexity; let us estimate it by taking a closer look at 3 equivalent but finer-grained propositions...’) and claiming each is max 99%, since 0.99^6=0.941.
With your 5 premises, that’d be starting with 95%, and then I can knock it down to 90% by splitting each premise which I could do very easily and have already implied in my first criticism about the self-improvement premise.
You can do this quite easily with cryonics as well—one attempt I saw included transportation to the hospital and used no probability >99%! Needless to say, the person concluded cryonics was a ludicrously bad idea.
It’s a strange kind of analysis that only allows the final probability to get smaller and smaller and smaller...
(Obviously as a violation of Cox’s theorems—by putting upper bounds on probabilities—this lets us get Dutchbooked.)
It wasn’t my intention to double-count probabilities. As far as what I wrote suggested that, I am simply wrong. My intention was to show that risks from AI are not as likely as its logical possibility and not as likely as its physical possibility. My intention was to show that there are various premises that need to be true, each of which introduces an additional probability penalty.
Then you should decompose it like this:
P(FOOM) = P(premise_n | premises_1..n-1) * P(premise_n-1 | premises_1..n-2) * … * P(premise_2 | premise_1) * P(premise_1)
Then you’re precisely measuring the additional probability penalty introduced. And if premise PX implies premise PY, you throw out PY for simplicity. If you can give an upper bound on P(PX) then you gave exactly the same upper bound P(PX & PY). You can’t make it stronger by reordering and writing P(PX & PY) = PY * (PX | PY) and then saying ‘but PY doesn’t imply PX so there’.
Talking about the conjunctive fallacy looks disingenuous when the conjuncts have strong dependencies.
I am sick of being accused of being disingenuous, using dark arts and countless other things like asking “rhetorical questions”. Are people really that incapable of seeing that I might simply lack the necessary training? Concluding that all I am saying is therefore just wrong is then making me use emotionally loaded language.
All those accusations rather look incredible sad. As if those people are just pissed off that someone tried to criticize their most cherished ideas but they don’t know what to say other than ridiculing the opponent based on his inexperience.
Sorry, I didn’t mean to ridicule you. I’m not annoyed be the fact that you’re criticizing—if I’m annoyed at all (does ‘someone is WRONG on the Internet!’ syndrome count as annoyance?). I wasn’t bothered by your criticisms of SI when you started posting them. But since then you’ve been going at it, repeating the same arguments over and over again.
You’re trying to create something out of nothing here. Currently available arguments about intelligence explosion are simple. There’s no deep math in them (and that’s a problem for sure but it cuts both ways — the SIAI don’t have a mathy model of intelligence explosion, you don’t have mathy arguments that recursive self-improvement will run into fundamental limitations). People are moved by those arguments to various extents. And that’s it. We’re done. Someone has to come up with a novel insight that will shed additional light on the issue. Until then, people won’t change their minds by being exposed to the same arguments even if they come with a brand new rhetorical packaging, heretofore unseen decomposition into bullet points, and a sprinkling of yet-unseen cool quotations.
People will change their minds by being exposed to new background knowledge that isn’t a directly about intelligence explosion but causes them to see existing arguments in new light. The sequences are a likely example of that. They will also change their minds for epistemologically insane reasons like social pressure. Both those factors are hard to affect and writing posts on LessWrong seems like one of the worst ways to go about it.
No one likes being told the same thing over and over again in an insistent tone of voice. If you do that, people will get frustrated and want to criticize you. If you give in to your intuitive feeling that you need to rephrase just a little bit and this time they will surely see the light, then you will eventually rephrase your way to bullshit and give those frustrated people ample opportunity to poke holes in your arguments.
Using somewhat different language this is exactly what you declare about yourself. Those things which you describe so casually as your own preferred behaviors are seen by those with a lesswrong mindset as disengenuity and the abuse of the dark arts. That isn’t necessarily a bad thing—you’d fit right in at MENSA for example, aside from the entry requirement I suppose—it just isn’t received well on lesswrong.
Was that some sort of a dig at Mensa, or XiXiDu, or both ? I know next to nothing about Mensa, so I feel like I’m missing the context here… Aren’t they just a bunch of guys who solve IQ tests as a hobby ?
Neither, more of a mild compliment combined with an acknowledgement that the lesswrong way is not the only way—or even particularly common.
It is not dark arts if you are honest about what you are doing.
What I am often doing is exploring various viewpoints by taking the position of someone who would be emotionally attached to it and convinced about it. I also use the opponents arguments against the opponent if it shows that it cuts both ways. I don’t see why that would be a problem, especially since I always admitted that I am doing that. See for example this comment from 2010.
That’s absolutely false. The terror management theory people, for example, discovered that mortality salience still kicks in even if you tell people that you’re going to expose them to something in order to provoke their own feeling of mortality.
EDIT: The paper I wanted to cite is still paywalled, afaik, but the relevant references are mostly linked in this section of the Wikipedia article. The relevant study is the one where the threat was writing about one’s feelings on death.
Okay. I possibly mistakenly assumed that the only way I could get answers is to challenge people directly and emotionally. I didn’t expect that I could just ask how people associated with SI/LW could possible believe what they believe and get answers. I tried, but it didn’t work.
It would be helpful if you summarised the premises in a short list. At the moment one has to do a lot of scrolling.
Edit: Actually, I think it would be a very good idea; their not being writen out together makes it easy to miss the fact that they’re not all necessary, some imply others, and that they basically don’t cut reality at its joints. You assert that these are all necessary and logically separate premises. Yet P4 is clearly not necessary for FOOM—something not being the default outcome does not mean it will not happen. P3 implies P2, and P2 implies P1. And P5 is clearly not necessary either—FOOM could occur in a thousand years time.
And again with the second set of premises—they are clearly not distinct, and not all necessary. For example,
P6 - SIAI will solve FAI
is not necessary; they might succeed by preventing anyone else from developing GAI.
P7 SIAI does not increase risks from AI.
If you mean net, then yes. But otherwise, it’s perfectly possible that they might speed up UFAI and AI, and yet still be a good thing, if the latter outweighs the former.
and
P9 It makes sense to support SIAI at this time
is the conclusion of the argument! This premise alone is sufficient—what are the others doing here? The only motivation I can see here is to make the conclusion seem artificially conjunctive.
Basically, as far as I can see you’ve chosen to split up factors so as to make the case seem more conjunctive, whilst ignoring those that make it disjunctive. Without any argument as to why the partition into overlapping, non-necessary factors you’ve given is the right one, this post reads more like rhetoric than analysis.
All this is quite appart from your actual arguments that these premises are unlikely.
That is actually my argument against a lot of philosophy; arguments embedded in a lot of prose are unnecessarily hard to follow. Arguments, at least ones that you actually expect to be capable of changing someone’s mind, should be presented as clearly and schematically as possible. Otherwise it looks a lot like “baffle them with bullshit.”
Reading this made my brain hurt. It’s a pile of false analogies that ignores the best arguments disagreeing with it, which is particularly ironic in light of the epigraph. (I’m thinking of Chalmers specifically, but really you can take your pick.)
I’m tempted to go through and point out every problem with this post, but I noticed at least a dozen on my first read-through and I just don’t have the time.
Posts arguing against the LW orthodoxy deserve disproportional attention and consideration to combat groupthink, but this is just too wrong for me to tolerate.
What I take a critical look at and what I write about in public are two very, very different things. Your audience is more heterogeneous than you might think.
You (XiXiDu) don’t seem to think that intelligent machines are likely to be that big a deal. The switch to an engineered world is likely to be the biggest transformation of the planet since around the evolution of sex—or the last genetic takeover. It probably won’t crash civilisation—or kill all humans—but it’s going to be an enormous change, and the deatils of how it goes down could make a big difference to everyone. I do sometimes wonder whether you get that.
Good post.
You seem to excessively focus on recursive self-improvement to the exclusion of other hard takeoff scenarios, however. As Eliezer noted,
That post mentions several other hard takeoff scenarios, e.g.:
Even if an AI’s self-improvement efforts quickly hit a wall, a small number of crucial optimizations, or the capture of a particular important resource, will provide it a massive intelligence advantage over humans. (Has evolutionary precedent in that the genetic differences between humans and chimps are relatively small. )
Parallel hardware overhang: if there’s much more hardware available than it takes to run an AI, an AI could expand itself and thus become more intelligent by simply “growing a bigger brain”, or create an entire society of co-operating AIs.
Serial hardware overhang: an AI running on processors with more serial speed than neurons could be able to e.g. process longer chains of inference instead of relying on cache lookups.
(Also a couple more, but I found those a little vague and couldn’t come up with a good way to summarize them in a few of sentences.)
Ah, thanks for making this point—I notice I’ve recently been treating “recursive self improvement and “hard takeoff” as more or less interchangeable concepts. I don’t think I need to update on this, but I’ll try and use my language more carefully at least.
Thanks. I will review those scenarios. Just some quick thoughts:
On first sight this sounds suspicious. The genetic difference between a chimp and a human amounts to about ~40–45 million bases that are present in humans and missing from chimps. And that number is irrespective of the difference in gene expression between humans and chimps. So it’s not like you’re adding a tiny bit of code and get a superapish intelligence.
The argument from the gap between chimpanzees and humans is interesting but can not be used to extrapolate onwards from human general intelligence. It is pure speculation that humans are not Turing complete and that there are levels above our own. That chimpanzees exist, and humans exist, is not a proof for the existence of anything that bears, in any relevant respect, the same relationship to a human that a human bears to a chimpanzee.
Humans can process long chains of inferences with the help of tools. The important question is if incorporating those tools into some sort of self-perception, some sort of guiding agency, is vastly superior to humans using a combination of tools and expert systems.
In other words, it is not clear that there does exist a class of problems that is solvable by Turing machines in general, but not by a combination of humans and expert systems.
If an AI that we invented can hold a complex model in its mind, then we can also simulate such a model by making use of expert systems. Being consciously aware of the model doesn’t make any great difference in principle to what you can do with the model.
Here is what Greg Egan has to say about this in particular:
The quote from Egan would seem to imply that for (literate) humans, too, working memory differences are insignificant: anyone can just use pen and paper to increase their effective working memory. But human intelligence differences do seem to have a major impact on e.g. job performance and life outcomes (e.g. Gottfredson 1997), and human intelligence seems to be very closely linked to—though admittedly not identical with—working memory measures (e.g. Oberauer et al. 2005, Oberauer et al. 2008).
I believe that what he is suggesting is that if you reached a certain plateau then intelligence hits diminishing returns. Would Marilyn vos Savant be proportionally more likely to take over the world, if she tried to, than a 115 IQ individual?
Some anecdotal evidence:
Is there evidence that a higher IQ is useful beyond a certain level? The question is not just if it is useful but if it would be worth the effort it would take to amplify your intelligence to that point given that your goal was to overpower lower IQ agent’s. Would a change in personality, more data, a new pair of sensors or some weapons maybe be more useful? If so, would an expected utility maximizer pursue intelligence amplification?
(A marginal note, bigger is not necessarily better.)
I upvoted for the anecdote, but remember that you’re referring to von Neumann, who invented both the basic architecture of computers and the self-replicating machine. I am not qualified to judge whether or not those are as original as relativity, but they are certainly big.
Sure. She’s demonstrated that she can communicate successfully with millions and handle her own affairs quite successfully, generally winning at life. This is comparable to, say, Ronald Reagan’s qualifications. I’d be quite unworried in asserting she’d be more likely to take over the world than a baseline 115 person.
Surely humans are Turing complete. I don’t think anybody disputes that.
We know that capabilities extend above our own in all the realms where machines already outstrip our capabilities—and we have a pretty good idea what greater speed, better memory and more memory would do.
Agree with your basic point, but a nit-pick: limited memory and speed (heat death of the universe, etc) put many neat Turing machine computations out of reach of humans (or other systems in our world) barring new physics.
Sure: I meant in the sense of the “colloquial usage” here:
Thanks for working this up.
However, it leads me to thinking about a modest FOOM. What’s the least level of intelligence needed for a UFAI to be an existential risk? What’s the least needed for it to be extremely deadly, even if not an existential risk?
What makes you think that it takes a general intelligence? Automatic scientists, with well-defined goals, that can brute-force discoveries on hard problem in bio and nanotech could enable unfriendly humans to wreck havoc and control large groups of people. If we survive that, which I think is the top risk rather than GAI, then we might at some point be able to come up with an universal artificial intelligence.
Think about it this way. If humans figure out how to create some sort of advanced narrow AI that can solve certain problems in a superhuman way, why would they wait and not just assign it directly to solving those problems?
The problem is that you can’t make such narrow AI’s “friendly”, because they are tools and not agents. Tools used by unfriendly humans.
Luckily there is a way to impede the consequences of that development and various existential risks at once.
What we should be working on is a global sensor network by merging various technologies like artificial noses, lab on a chip technology, DNA Sequencing To Go etc.
Such a sensor network could be used to detect various threats like nuclear terrorism with dirty bombs, venomed water or biological pathogens early on and alert authorities or nearby people.
You could work with mobile phone companies to incorporate those sensors into their products. Companies like Apple would profit from having such sensors in their products by extending their capabilities. This would not only allow the mass production but would also spread the sensors randomly.
You might also work together with the government who is always keen to get more information. All it would then take is an app!
The analysis of the data could actually be done by the same gadgets that employ the sensors, a public computing grid.
This isn’t science fiction, it can actually be done. The technology is coming quickly.
And best of all, it doesn’t just protect us against various risks. Such sensors could be used to detect all kinds of health problems or stress levels and automatically call for help.
I was with you up until this sentence. Really, we can make a global sensor network today ? A network that would detect all conceivable threats everywhere ? This sounds just a tad unrealistic to me, though not logically impossible at some point in the future.
Thanks—I’ve read the bullet points and it looks like a really good summary (apologies for skimming—I’ll read it in more detail when I have time).
Just a few minor points:
The P(FOOM) calculation appears to be entirely independent of the P(CHARITY) calculation. Should these be made into separate documents? Or should it be made clearer which factors are common to FOOM and CHARITY? (e.g. P5 would appear to be correlated with P9).
In P6, I’m taking “SIAI” to mean a kind of generalized SIAI (i.e. it doesn’t have to be this specific team of people who solve FAI, what we’re interested in is to what extent a donation to this organization increases the probability that someone will solve FAI)
P7 and P8: I’m not sure which risk factors go into P7 and which go into P8. I’d have listed them as one top-level point with a bunch of sub-points.
P7 and P8: I think that sane SIAI supporters believe that supporting SIAI reduces some risk pathways while increasing others. The standard is not “it mustn’t increase any risks” but rather “the expected positives must outweigh the negatives”
Also if the standard is not “a worthwhile charity” but “the best charity”, it would be worth adding a P10: no other charity provides higher expected marginal value. Meta-level charities that focus on building the rational altruism movement are at least a candidate here.
I wanted to show that even if you assign a high probability to the possibility of risks from AI due to recursive self-improvement, it is still questionable if SIAI is the right choice or if now is the time to act.
As I wrote at the top, it was a rather quick write-up and I plan to improve it. I can’t get myself to work on something like this for very long. It’s stupid, I know. But I can try to improve things incrementally. Thanks for your feedback.
That’s a good point. SIAI as an organisation that makes people aware of the risk. But from my interview series it seemed like that a lot of AI researchers are aware of it to the point of being bothered.
It isn’t optimal. It is kind of hard to talk about premises that appear to be the same from a superficially point of view. But from a probabilistic point of view it is important to separate them into distinct parts to make clear that there are things that need to be true in conjunction.
That problem is incredibly mathy and given my current level of education I am happy that people like Holden Karnofsky tackle that problem. The problem being that we get into the realm of Pascal’s mugging here where vast utilities outweigh tiny probabilities. Large error bars may render such choices moot. For more, see my post here.
If I understand you correctly, you are saying this: “Don’t bother with this superintelligence risk, for it is incredibly tiny.”
A bold statement. Too bold for a potentially disastrous chain of events, which you assure us it’s just impossible.
No, not really. I am not saying that because GiveWell says that the Against Malaria Foundation is the number #1 charity, treating children for parasite infections in sub-Saharan Africa should be ignored.
This is a delicate problem and if it was up to me to allocate the resources of the world then existential risk researchers, including SIAI, would receive their share of funding. But if I could only choose one cause, either SIAI or something else, then I wouldn’t choose SIAI.
My opinion on the topic is highly volatile though. There have been moments when I thought that SIAI is best choice when it comes to charitable giving. There has been a time when I was completely sure that a technological singularity will happen soon. Maybe I will change my mind again. I suggest everyone to research the topic themselves.
This I agree. I won’t buy the whole package from the SIAI, I won’t even donate them under the current conditions.
But I see some of their points as extremely important and I am happy that they exist and do what they do.
This is wrong because the premises aren’t independent. It’s actually this:
P(P1) P(P2|P1) P(P3|P2∧P1) P(P4|P3∧P2∧P1) P(P5|P4∧P3∧P2∧P1)
Not only there has to be UFAI risk, the FAI development must reduce the risk, which to me looks like the most shaky of the propositions. A buggy FAI that doesn’t break itself somehow is for certain unfriendly (e.g. it can want to euthanize you to end your suffering, or to cut apart your brain into 2 hemispheres to satisfy each hemisphere’s different desires, or something much more bizarre), while some random AI out of AI design space may e.g. typically wirehead everything except curiosity, and then it’d just keep us in a sort of wildlife preserve.
Note: try to avoid just world fallacy. Just because you work harder to make friendlier AI doesn’t necessarily make result friendlier. The universe doesn’t grade for effort.
We humans do a lot of wireheaded stuff. Fiction, art, MSG in food, porn, cosmetic implants… we aren’t doing it literally with a wire into the head, but we find creative ways to satisfy goals… maybe even the desire to build AI itself is a result of breaking the goal system. And the wireheadedness makes us tolerate other forms of life instead of setting on to exterminate everything as we already would have if we still pursued some reproductive goals.
Hell, we even wirehead our curiosity and quest for knowledge (see religions), entirely internally by breaking the goal system with self suggestion.
re: Benoit Mandelbrot and fractals, the fractals are way, way older. The actual study of them had to wait until computers.
Fantastic post. This sets a new standard for “SIAI skepticism”. Dialectically it should be very useful as people try to rebut it at the same level of detail. I think you shouldn’t mess with it too much now, as it may become a reference point.