Publishers choose from a wide range of authors who want to make predictions. They choose those that are most exciting. These come from the “fast change” and “strange assumptions” end of the distribution. Any prediction you actually hear about is therefore likely to be wrong.
Phil_Goetz6
The rapidity of evolution from chimp to human is remarkable, but you can infer what you’re trying to infer only if you believe evolution reliably produces steadily more intelligent creatures. It might be that conditions temporarily favored intelligence, leading to humans; our rapid rise is then explained by the anthropic principle, not by universal evolutionary dynamics.
Knowledge = all that actual science, engineering, and general knowledge accumulation we did = integral of cognition+metaknowledge(current knowledge) over time, where knowledge feeds upon itself in what seems to be a roughly exponential process
Knowledge feeds on itself only when it is continually spread out over new domains. If you keep trying to learn more about the same domain—say, to cure cancer, or make faster computer chips—you get logarithmic returns, requiring an exponential increase in resources to maintain constant output. (IIRC it has required exponentially-increasing capital investments to keep Moore’s Law going; the money will run out before the science does.) Rescher wrote about this in the 1970s and 1980s.This is important because it says that, if an AI keeps trying to learn how to improve itself, it will get only logarithmic returns.
When you fold a complicated, choppy, cascade-y chain of differential equations in on itself via recursion, it should either flatline or blow up. You would need exactly the right law of diminishing returns to fly through the extremely narrow soft takeoff keyhole.
This is the most important and controversial claim, so I’d like to see it better-supported. I understand the intuition; but it is convincing as an intuition only if you suppose there are no negative feedback mechanisms anywhere in the whole process, which seems unlikely.
A number of people are objecting to Eliezer’s claim that the process he is discussing is unique in its FOOM potential, proposing other processes that are similar. Then Eliezer says they aren’t similar.
Whether they’re similar enough depends on the analysis you want to do. If you want to glance at them and come up with yes or no answer regarding FOOM, then none of them are similar. A key difference is that these other things don’t have continual halving of the time per generation. You can account for this when comparing results, but I haven’t seen anyone do this.
But some things are similar enough that you can gain some insights into the AI FOOM potential by looking at them. Consider the growth of human societies. A human culture/civilization/government produces ideas, values, and resources used to rewrite itself. This is similar to the AI FOOM dynamics, except with constant and long generation times.
To a tribesman contemplating the forthcoming culture FOOM, it would look pretty simple: Culture is about ways for your tribe to get more land than other tribes.
As culture progressed, we developed all sorts of new goals for it that the tribesman couldn’t have predicted.
Analogously, our discussion of the AI FOOM supposes that the AI will not discover new avenues to pursue other than intelligence, that soak up enough of the FOOM to slow down the intelligence part of the FOOM considerably. (Further analysis of this is difficult since we haven’t agreed what “intelligence” is.)
Another lesson to learn from culture has to do with complexity. The tribesman, given some ideas of what technology and government would do, would suppose that it would solve all problems. But in fact, as cultures grow more capable, they are able to sustain more complexity; and so our problems get more and more complicated. The idea that human stupidity is holding us back, and AIs will burst into exponential territory once they shake free of these shackles:
I suspect that human economic growth would naturally tend to be faster and somewhat more superexponential, if it were not for the negative feedback mechanism of governments and bureaucracies with poor incentives, that both expand and hinder whenever times are sufficiently good that no one is objecting strongly enough to stop it
is like that tribesman thinking good government will solve all problems. Systems—societies, governments, AIs—expand to the limits of complexity that they can support; at those limits, actions have unintended consequences and agents have not quite enough intelligence to predict them or agree on them, and in efficiency and “stupidity”—relative stupidity—lives on.I’ll respond to Eliezer’s response to my response later today. Short answer: 1. Diminishing returns exist and are powerful. 2. This isn’t something you can eyeball. If you want to say FOOM is probable; fine. If you want to say FOOM is almost inevitable, I want to see equations worked out with specific numbers. You won’t convince me with handwaving, especially when other smart people are waving their hands and reaching different conclusions.
I wrote:
Analogously, our discussion of the AI FOOM supposes that the AI will not discover new avenues to pursue other than intelligence, that soak up enough of the FOOM to slow down the intelligence part of the FOOM considerably.
What I wish I’d said is: What percentage of the AI’s efforts will go into algorithm, architecture, and hardware research?At the start, probably a lot; so this issue may not be important wrt FOOM and humans.
“Sexual selection is at the root of practically all the explanations for the origin of our large brains.”
Ooh, you triggered one of my cached rants.
Practically all of those explanations start by saying something like, “It’s a great mystery how humans got so smart, since you don’t need to be that smart to gather nuts and berries.”
And that shows tremendous ignorance of how much intelligence is needed to be a hunter-gatherer. (Much more than is needed to be a modern city-dweller.) Most predators have a handful of ways of catching prey; primitive humans have thousands. Just enumerating different types of snares and traps used would bring us over 100.
“For “specific numbers”, for example, look at the well-documented growth of the computer industry since the 1950s.”
You would need to show how to interpret those numbers applied to the AI foom.
I’d rather see a model for AI foom built from the ground up, and ranges of reasonable values posited, and validated in some way.
This is a lot of work, but after several years working on the problem, it’s one that ought to have a preliminary answer.
“All these complications is why I don’t believe we can really do any sort of math that will predict quantitatively the trajectory of a hard takeoff. You can make up models, but real life is going to include all sorts of discrete jumps, bottlenecks, bonanzas, insights—and the “fold the curve in on itself” paradigm of recursion is going to amplify even small roughnesses in the trajectory.”
Wouldn’t that be a reason to say, “I don’t know what will happen”? And to disallow you from saying, “An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely”?
If you can’t make quantitative predictions, then you can’t say that the foom might take an hour or a day, but not six months.
A lower-bound (of the growth curve) analysis could be sufficient to argue the inevitability of foom.
I agree there’s a time coming when things will happen too fast for humans. But “hard takeoff”, to me, means foom without warning. If the foom doesn’t occur until the AI is smart enough to rewrite an AI textbook, that might give us years or decades of warning. If humans add and improve different cognitive skills to the AI one-by-one, that will start a more gently-sloping RSI.
Eliezer: So really, the whole hard takeoff analysis of “flatline or FOOM” just ends up saying, “the AI will not hit the human timescale keyhole.” From our perspective, an AI will either be so slow as to be bottlenecked, or so fast as to be FOOM.
But the AI is tied up with the human timescale at the start. All of the work on improving the AI, possibly for many years, until it reaches very high intelligence, will be done by humans. And even after, it will still be tied up with the human economy for a time, relying on humans to build parts for it, etc. Remember that I’m only questioning the trajectory for the first year or decade.(BTW, the term “trajectory” implies that only the state of the entity at the top of the heap matters. One of the human race’s backup plans should be to look for a niche in the rest of the heap. But I’ve already said my piece on that in earlier comments.)
Thomas: Even if it is wrong—I think it is correct—it is the most important thing to consider.
I think most of us agree it’s possible. I’m only arguing that other possibilities should also be considered. It would be unwise to adopt a strategy that has a 1% chance of making 90%-chance situation A survivable, if that strategy will make the otherwise-survivable 10%-chance situation B deadly.
I designed, with a co-worker, a cognitive infrastructure for DARPA that is supposed to let AIs share code. I intended to have cognitive modules be web services (at present, they’re just software agents). Every representation used was to be evaluated using a subset of Prolog, so that expressions could be automatically converted between representations. (This was never implemented; nor was ontology mapping, which is really hard and would also be needed to translate content.) Unfortunately, my former employer didn’t let me publish anything on it. Also, it works only with symbolic AI.
It wouldn’t change the picture Eliezer is drawing much even if it worked perfectly, though.
on any sufficiently short timescale, progress should locally approximate an exponential because of competition between interest rates (even in the interior of a single mind).
Huh?
“The issue is that simulating a computers design require a lot of computational power. The advances made in going from 65nm to 45nm now moving to 32nm were enabled by computers that could better simulate the designs without todays computers it would be hard to design the fabrication systems or run the fabrication system for the future processors.”
I believe (strongly) that the bottleneck is figuring out how to make 45nm and 32nm circuits work reliably. If you learn how to do 32nm, you can probably get speedup just by re-using the same design you used at 45nm.
How is this different from saying,
“For a long time, many different parties and factions in AI, adherent to more than one ideology, have been trying to build AI without understanding consciousness. Unfortunate habits of thought will already begin to arise, as soon as you start thinking of ways to create Artificial Intelligence without having to penetrate the mystery of consciousness. Instead of all this mucking about with neurons and neuroanatomy and population encoding and spike trains, we should be facing up to the hard problem of understanding what consciousness is.”
Eliezer: I was making a parallel. I didn’t mean “how are these different”; I really meant, “This statement below about consciousness is wrong; yet it seems very similar to Eliezer’s post. What is different about Eliezer’s post that would make it not be wrong in the same way?”
That said, we don’t know what consciousness is, and we don’t know what intelligence is; and both occur in every instance of intelligence that we know of; and it would be surprising to find one without the other even in an AI; so I don’t think we can distinguish between them.
I may be wrong about Newcomb’s paradox.
You could say that embracing timeless decision theory is a global meta-commitment, that makes you act as if you made commitment in all the situations where you benefit from having made the commitment.
I think this is correct.It’s perplexing: This seems like a logic problem, and I expect to make progress on logic problems using logic. I would expect reading an explanation to be more helpful than having my subconscious mull over a logic problem. But instead, the first time I read it, I couldn’t understand it properly because I was not framing the problem properly. Only after I suddenly understood the problem better, without consciously thinking about it, was I able to go back, re-read this, and understand it.
But if you’re going to bother visualizing the future, it does seem to help to visualize more than one way it could go, instead of concentrating all your strength into one prediction. So I try not to ask myself “What will happen?” but rather “Is this possibility allowed to happen, or is it prohibited?”
I thought that you were changing your position; instead, you have used this opening to lead back into concentrating all your strength into one prediction.
I think this characterizes a good portion of the recent debate: Some people (me, for instance) keep saying “Outcomes other than FOOM are possible”, and you keep saying, “No, FOOM is possible.” Maybe you mean to address Robin specifically; and I don’t recall any acknowledgement from Robin that foom is >5% probability. But in the context of all the posts from other people, it looks as if you keep making arguments for “FOOM is possible” and implying that they prove “FOOM is inevitable”.
A second aspect is that some people (again, eg., me) keep saying, “The escalation leading up to the first genius-level AI might be on a human time-scale,” and you keep saying, “The escalation must eventually be much faster than human time-scale.” The context makes it look as if this is a disagreement, and as if you are presenting arguments that AIs will eventually self-improve themselves out of the human timescale and saying that they prove FOOM.
I don’t think it did help, though. I think I failed to comprehend it. I didn’t file it away and think about it; I completely missed the point. Later, my subconscious somehow changed gears so that I was able to go back and comprehend it. But communication failed.
Buddhists say that great truths can’t be communicated; they have to be experienced, only after which you can understand the communication. This was something like that. Discouraging.
Eliezer: “and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever).”
Why do the values freeze? Because there is no more competition? And if that’s the problem, why not try to plan a transition from pre-AI to an ecology of competing AIs that will not converge to a singleton? Or spell out the problem clearly enough that we can figure whether one can achieve a singleton that doesn’t have that property?
(Not that Eliezer hasn’t heard me say this before. I made a bit of a speech about AI ecology at the end of the first AGI conference a few years ago.)
Robin: “In a foom that took two years, if the AI was visible after one year, that might give the world a year to destroy it.”
Yes. The timespan of the foom is important largely because it changes what the AI is likely to do, because it changes the level of danger that the AI is in and the urgency of its actions.
Eliezer: “When I try myself to visualize what a beneficial superintelligence ought to do, it consists of setting up a world that works by better rules, and then fading into the background.”
There are many sociological parallels between Eliezer’s “movement”, and early 20th-century communism.
Eliezer: “I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.”
I wonder if you’re thinking that I meant that. You can see that I didn’t in my first comment on Visions of Heritage. But I do think you’re going one level too few meta. And I think that CEV would make it very hard to escape the non-meta philosophies of the programmers. It would be worse at escaping them than the current, natural system of cultural evolution is.
Numerous people have responded to some of my posts by saying that CEV doesn’t restrict the development of values (or equivalently, that CEV doesn’t make AIs less free). Obviously it does. That’s the point of CEV. If you’re not trying to restrict how values develop, you might as well go home and watch TV and let the future spin out of control. One question is where “extrapolation” fits on a scale between “value stasis” and “what a free wild-type AI would think of on its own.” Is it “meta-level value stasis”?
I think that evolution and competition have been pretty good at causing value development. (That’s me going one more level meta.) Having competition between different subpopulations with different values is a key part of this. Taking that away would be disastrous.
Not to mention the fact that value systems are local optima. If you’re doing search, it might make sense to average together some current good solutions and test the results out, in competition with the original solutions. It is definitely a bad idea to average together your current good solutions and replace them with the average.
Eliezer: “Tim probably read my analysis using the self-optimizing compiler as an example, then forgot that I had analyzed it and thought that he was inventing a crushing objection on his own. This pattern would explain a lot of Phil Goetz too.”
No; the dynamic you’re thinking of is that I raise objections to things that you have already analyzed, because I think your analyis was unconvincing. Eg., the recent Attila the Hun / Al Qaeda example. The fact that you have written about something doesn’t mean you’ve dealt with it satsifactorily.
It would have been better of me to reference Eliezer’s Al Qaeda argument, and explain why I find it unconvincing.
Vladimir:
Phil, in suggesting to replace an unFriendly AI that converges on a bad utility by a collection of AIs that never converge, you are effectively trying to improve the situation by injecting randomness in the system.
You believe evolution works, right?You can replace randomness only once you understand the search space. Eliezer wants to replace the evolution of values, without understanding what it is that that evolution is optimizing. He wants to replace evolution that works, with a theory that has so many weak links in its long chain of logic that there is very little chance it will do what he wants it to, even supposing that what he wants it to do is the right thing to do.
Vladimir:
Your perception of lawful extrapolation of values as “stasis” seems to stem from intuitions about free will.
That’s a funny thing to say in response to what I said, including: ‘One question is where “extrapolation” fits on a scale between “value stasis” and “what a free wild-type AI would think of on its own.”’ It’s not that I think “extrapolation” is supposed to be stasis; I think it may be incoherent to talk about an “extrapolation” that is less free than “wild-type AI”, and yet doesn’t keep values out of some really good areas in value-space. Any way you look at it, it’s primates telling superintelligences what’s good.As I just said, clearly “extrapolation” is meant to impose restrictions on the development of values. Otherwise it would be pointless.
Vladimir:
it could act as a special “luck” that in the end results in the best possible outcome given the allowed level of interference.
Please remember that I am not assuming that FAI-CEV is an oracle that magically works perfectly to produce the best possible outcome. Yes, an AI could subtly change things so that we’re not aware that it is RESTRICTING how our values develop. That doesn’t make it good for the rest of all time to be controlled by the utility functions of primates (even at a meta level).Here’s a question whose answer could diminish my worries: Can CEV lead to the decision to abandon CEV? If smarter-than-humans “would decide” (modulo the gigantic assumption CEV makes that it makes sense to talk about what “smarter than humans would decide”, as if greater intelligence made agreement more rather than less likely—and, no, they will not be perfect Bayesians) that CEV is wrong, does that mean an AI guided by CEV would then stop following CEV?
If this is so, isn’t it almost probability 1 that CEV will be abandoned at some point?
suggests that somebody has read Ender’s Game too many times. These are three gigantic research projects. I think he should work on #2 or #3.how to build an AI
how to make an AI stay within a specified range of behavior, and
what an AI ought to do
Not doing #1 would mean that it actually matters that he convince other people of his ideas.
I think that #3 is really, really tricky. Far beyond the ability of any one person. This blog may be the best chance he’ll have to take his ideas, lay them out, and get enough intelligent criticism to move from the beginnings he’s made, to something that might be more useful than dangerous. Instead, he seems to think (and I could be wrong) that the collective intelligence of everyone else here on Overcoming Bias is negligible compared to his own. And that’s why I get angry and sometimes rude.
Generalizing from observations of points at the extremes of distributions, we can say that when we find an effect many standard deviations away from the mean, its position is almost ALWAYS due more to random chance than to the properties underlying that point. So when we observe a Newton or an Einstein, the largest contributor to their accomplishments was not their intellect, but random chance. So if you think you’re relying on someone’s great intellect, you’re really relying on chance.