In Superintelligence, Bostrom defines a “takeoff” as the transition “from human-level intelligence to superintelligence.” This seems like a poor definition for several reasons: “human-level intelligence” is a bad concept, and “superintelligence” is ambiguous in Bostrom’s book between “strong superintelligence” (“a level of intelligence vastly greater than contemporary humanity’s combined intellectual wherewithal”) and “[weak?] superintelligence” (“greatly [exceeding] the cognitive performance of [individual?] humans in virtually all domains of interest”).
Moreover, neither of these thresholds is strategically important/relevant: “superintelligence” is too high and anthropocentric a bar for talking about seed AGI, and is too low a bar for talking about decisive strategic advantage; whereas “strong superintelligence” is just a really weird/arbitrary/confusing bar when the thing we care about is DSA.
More relevant thresholds on my view are things like “is it an AGI yet? can it, e.g., match the technical abilities of an average human engineer in at least one rich, messy real-world scientific area?” and “is it strong enough to prevent any competing AGI systems from being deployed in the future?”
All of this is to say that “takeoff” in Bostrom’s sense may not be the most helpful term. That said, Bostrom defines a fast takeoff as one that takes “minutes, hours, or days,” a moderate takeoff as one that takes “months or years,” and a slow takeoff as one that takes “decades or centuries.”
A further problem with applying these definitions to the present discussion is that the question Grace/Hanson/Christiano care about is often “how well can humanity, or human institutions, or competing AI projects, keep up with an AGI project?”, but Bostrom’s definitions of “superintelligence” are unclear about whether they’re assuming some static threshold (e.g., ‘capability of humans in 2014’ or ‘capability of human when the first AGI begins training’) versus a moving threshold that can cause an AGI to fall short of “superintelligence” because other actors are keeping pace.
The place to situate the disagreement for mainstream skeptics of what Eliezer calls “rapid capability gain” might be something like: “Once we have AGI, is it more likely to take 2 subjective years to blow past human scientific reasoning in the way AlphaZero blew past human chess reasoning, or 10 subjective years?” I often phrase the MIRI position along the lines of “AGI destroys or saves the world within 5 years of being developed”.
That’s just talking in terms of widely held views in the field, though. I think that e.g. MIRI/Christiano disagreements are less about whether “months” versus “years” is the right timeframe, and more about things like: “Before we get AGI, will we have proto-AGI that’s nearly as good as AGI in all strategically relevant capabilities?” And the MIRI/Hanson disagreements are maybe less about months vs. years and more about whether AGI will be a discrete software product invented at a particular time and place at all.
In Superintelligence, Bostrom defines a “takeoff” as the transition “from human-level intelligence to superintelligence.” This seems like a poor definition for several reasons: “human-level intelligence” is a bad concept, and “superintelligence” is ambiguous in Bostrom’s book between “strong superintelligence” (“a level of intelligence vastly greater than contemporary humanity’s combined intellectual wherewithal”) and “[weak?] superintelligence” (“greatly [exceeding] the cognitive performance of [individual?] humans in virtually all domains of interest”).
Moreover, neither of these thresholds is strategically important/relevant: “superintelligence” is too high and anthropocentric a bar for talking about seed AGI, and is too low a bar for talking about decisive strategic advantage; whereas “strong superintelligence” is just a really weird/arbitrary/confusing bar when the thing we care about is DSA.
More relevant thresholds on my view are things like “is it an AGI yet? can it, e.g., match the technical abilities of an average human engineer in at least one rich, messy real-world scientific area?” and “is it strong enough to prevent any competing AGI systems from being deployed in the future?”
All of this is to say that “takeoff” in Bostrom’s sense may not be the most helpful term. That said, Bostrom defines a fast takeoff as one that takes “minutes, hours, or days,” a moderate takeoff as one that takes “months or years,” and a slow takeoff as one that takes “decades or centuries.”
A further problem with applying these definitions to the present discussion is that the question Grace/Hanson/Christiano care about is often “how well can humanity, or human institutions, or competing AI projects, keep up with an AGI project?”, but Bostrom’s definitions of “superintelligence” are unclear about whether they’re assuming some static threshold (e.g., ‘capability of humans in 2014’ or ‘capability of human when the first AGI begins training’) versus a moving threshold that can cause an AGI to fall short of “superintelligence” because other actors are keeping pace.
The place to situate the disagreement for mainstream skeptics of what Eliezer calls “rapid capability gain” might be something like: “Once we have AGI, is it more likely to take 2 subjective years to blow past human scientific reasoning in the way AlphaZero blew past human chess reasoning, or 10 subjective years?” I often phrase the MIRI position along the lines of “AGI destroys or saves the world within 5 years of being developed”.
That’s just talking in terms of widely held views in the field, though. I think that e.g. MIRI/Christiano disagreements are less about whether “months” versus “years” is the right timeframe, and more about things like: “Before we get AGI, will we have proto-AGI that’s nearly as good as AGI in all strategically relevant capabilities?” And the MIRI/Hanson disagreements are maybe less about months vs. years and more about whether AGI will be a discrete software product invented at a particular time and place at all.
I tend to agree with Robin that AGI won’t be a discrete product, though that’s much less confident.