That part of the interview with Paul was super interesting to me, because the following were previously claims I’d heard from Nate and Eliezer in their explanations of how they think about fast takeoff:
[E]volution [hasn’t] been putting a decent amount of effort into optimizing for general intelligence. [...]
‘I think if you optimize AI systems for reasoning, it appears much, much earlier.’
Ditto things along the lines of this Paul quote from the same 80K interview:
It’s totally conceivable from our current perspective, I think, that an intelligence that was as smart as a crow, but was actually designed for doing science, actually designed for doing engineering, for advancing technologies rapidly as possible—it is quite conceivable that such a brain would actually outcompete humans pretty badly at those tasks.
I think that’s another important thing to have in mind, and then when we talk about when stuff goes crazy, I would guess humans are an upper bound for when stuff goes crazy. That is we know that if we had cheap simulated humans, that technological progress would be much, much faster than it is today. But probably stuff goes crazy somewhat before you actually get to humans.
This is part of why I don’t talk about “human-level” AI when I write things for MIRI.
If you think humans, corvids, etc. aren’t well-optimized for economically/pragmatically interesting feats, this predicts that timelines may be shorter and that “human-level” may be an especially bad way of thinking about the relevant threshold(s).
There still remains the question of whether the technological path to “optimizing messy physical environments” (or “science AI”, or whatever we want to call it) looks like a small number of “we didn’t know how to do this at all, and now we do know how to do this and can suddenly take much better advantage of available compute” events, vs. looking like a large number of individually low-impact events spread out over time.
If no one event is impactful enough, then a series of numerous S-curves ends up looking like a smooth slope when you zoom out; and large historical changes are usually made of many small changes that add up to one big effect. We don’t invent nuclear weapons, get hit by a super-asteroid, etc. every other day.
MIRI thinks that the fact Evolution hasn’t been putting much effort into optimizing for general intelligence is a reason to expect discontinuous progress? Apparently, Paul’s point is that once we realize evolution has been putting little effort into optimizing for general intelligence, we realize we can’t tell much about the likely course of AGI development from evolutionary history, which leaves us in the default position of ignorance. Then, he further argues that the default case is that progress is continuous.
So far as I can tell, Paul’s point is that absent specific reasons to think otherwise, the prima facie case that any time we are trying hard to optimize for some criteria, we should expect the ‘many small changes that add up to one big effect’ situation.
Then he goes on to argue that the specific arguments that AGI is a rare case where this isn’t true (like nuclear weapons) are either wrong or aren’t strong enough to make discontinuous progress plausible.
From what you just wrote, it seems like the folks at MIRI agree that we should have the prima facie expectation of continuous progress, and I’ve read elsewhere that Eliezer thinks the case for recursive self-improvement leading to a discontinuity is weaker or less central than it first seemed. So, are MIRI’s main reasons for disagreeing with Paul down to other arguments (hence the switch from the intelligence explosion hypothesis to the general idea of rapid capability gain)?
I would think the most likely place to disagree with Paul (if not on the intelligence explosion hypothesis) would be if you expected the right combination of breakthroughs exceeds to a ‘generality threshold’ (or ‘secret sauce’ as Paul calls it) that leads to a big jump in capability, but inadequate achievement on any one of the breakthroughs won’t do.
Stuart Russell gives a list of the elements he thinks will be necessary for the ‘secret sauce’ of general intelligence in Human Compatible: human-like language comprehension, cumulative learning, discovering new action sets and managing its own mental activity. (I would add that somebody making that list 30 years ago would have added perception and object recognition, and somebody making it 60 years ago would have also added efficient logical reasoning from known facts). Let’s go with Russell’s list, so we can be a bit more concrete. Perhaps this is your disagreement:
An AI with (e.g.) good perception and object recognition, language comprehension, cumulative learning capability and ability to discover new action sets but a merely adequate or bad ability to manage its mental activity would be (Paul thinks) reasonably capable compared to an AI that is good at all of these things, but (MIRI thinks) it would be much less capable. MIRI has conceptual arguments (to do with the nature of general intelligence) and empirical arguments (comparing human/chimp brains and pragmatic capabilities) in favour of this hypothesis, and Paul thinks the conceptual arguments are too murky and unclear to be persuasive and that the empirical arguments don’t show what MIRI thinks they show. Am I on the right track here?
That part of the interview with Paul was super interesting to me, because the following were previously claims I’d heard from Nate and Eliezer in their explanations of how they think about fast takeoff:
Ditto things along the lines of this Paul quote from the same 80K interview:
This is part of why I don’t talk about “human-level” AI when I write things for MIRI.
If you think humans, corvids, etc. aren’t well-optimized for economically/pragmatically interesting feats, this predicts that timelines may be shorter and that “human-level” may be an especially bad way of thinking about the relevant threshold(s).
There still remains the question of whether the technological path to “optimizing messy physical environments” (or “science AI”, or whatever we want to call it) looks like a small number of “we didn’t know how to do this at all, and now we do know how to do this and can suddenly take much better advantage of available compute” events, vs. looking like a large number of individually low-impact events spread out over time.
If no one event is impactful enough, then a series of numerous S-curves ends up looking like a smooth slope when you zoom out; and large historical changes are usually made of many small changes that add up to one big effect. We don’t invent nuclear weapons, get hit by a super-asteroid, etc. every other day.
MIRI thinks that the fact Evolution hasn’t been putting much effort into optimizing for general intelligence is a reason to expect discontinuous progress? Apparently, Paul’s point is that once we realize evolution has been putting little effort into optimizing for general intelligence, we realize we can’t tell much about the likely course of AGI development from evolutionary history, which leaves us in the default position of ignorance. Then, he further argues that the default case is that progress is continuous.
So far as I can tell, Paul’s point is that absent specific reasons to think otherwise, the prima facie case that any time we are trying hard to optimize for some criteria, we should expect the ‘many small changes that add up to one big effect’ situation.
Then he goes on to argue that the specific arguments that AGI is a rare case where this isn’t true (like nuclear weapons) are either wrong or aren’t strong enough to make discontinuous progress plausible.
From what you just wrote, it seems like the folks at MIRI agree that we should have the prima facie expectation of continuous progress, and I’ve read elsewhere that Eliezer thinks the case for recursive self-improvement leading to a discontinuity is weaker or less central than it first seemed. So, are MIRI’s main reasons for disagreeing with Paul down to other arguments (hence the switch from the intelligence explosion hypothesis to the general idea of rapid capability gain)?
I would think the most likely place to disagree with Paul (if not on the intelligence explosion hypothesis) would be if you expected the right combination of breakthroughs exceeds to a ‘generality threshold’ (or ‘secret sauce’ as Paul calls it) that leads to a big jump in capability, but inadequate achievement on any one of the breakthroughs won’t do.
Stuart Russell gives a list of the elements he thinks will be necessary for the ‘secret sauce’ of general intelligence in Human Compatible: human-like language comprehension, cumulative learning, discovering new action sets and managing its own mental activity. (I would add that somebody making that list 30 years ago would have added perception and object recognition, and somebody making it 60 years ago would have also added efficient logical reasoning from known facts). Let’s go with Russell’s list, so we can be a bit more concrete. Perhaps this is your disagreement:
An AI with (e.g.) good perception and object recognition, language comprehension, cumulative learning capability and ability to discover new action sets but a merely adequate or bad ability to manage its mental activity would be (Paul thinks) reasonably capable compared to an AI that is good at all of these things, but (MIRI thinks) it would be much less capable. MIRI has conceptual arguments (to do with the nature of general intelligence) and empirical arguments (comparing human/chimp brains and pragmatic capabilities) in favour of this hypothesis, and Paul thinks the conceptual arguments are too murky and unclear to be persuasive and that the empirical arguments don’t show what MIRI thinks they show. Am I on the right track here?