Summary of why I think the post’s estimates are too low as estimates of what’s required for a system capable of seizing a decisive strategic advantage:
To be an APS-like system OmegaStar needs to be able to control robots or model real world stuff and also plan over billions, not hundreds of action steps.
Each of those problems adds on a few extra OOMs that aren’t accounted for in e.g. the setup for Omegastar (which can transfer learn across tens of thousands of games, each requiring thousands of action steps to win in a much less complicated environment than the real world).
You’d need something that can transfer learn across tens of thousands of ‘games’ each requiring billions of action steps, each one of which has way more sensory input to parse than StarCraft per time step.
When you correct Omegastar’s requirements by adding on (1) a factor for number of action steps needed to win a war Vs win a game of StarCraft, (2) a factor for the real world Vs StarCraft’s complexity of sensory input and. When you do this, the total requirement would look more like Ajeya’s reports.
I still get the intuition that OmegaStar would not just be a fancy game player! I find it hard to think about what it would be like—maybe good at gaming quite constrained systems or manipulating people?
Therefore, I think the arguments provide a strong case (unless scaling laws break—which I also think is fairly likely for technical reasons) for ‘something crazy happening by 2030’ but less strong a case for ‘AI takeover by 2030’
Summary of mine and Daniel’s disagreements:
(1) Horizon Length: Daniel thinks we’ll get a long way towards planning over a billion action steps ‘for free’ if we transfer learn over lots of games that take a thousand action steps each—so the first correction factor I gave is a lot smaller than it seems just by comparing the raw complexity of StarCraft Vs fighting a war
(2) No Robots: the complexity in sensory input difference doesn’t matter since the system won’t need to control robots [Or, as I should have also said, build robot-level models of the external world even if you’re not running the actuators yourself] - so the second correction factor isn’t an issue, because-
(3) Lower capability threshold: to take a DSA doesn’t require as many action steps as it seems or as many capabilities as often assumed. You can just do it by taking to people and over a smaller number of action steps than it would take to conquer the world yourself.
To me, it seems like Daniel’s view on horizon length reducing one of the upward corrections (1) is doing less total work than (2) and (3) in terms of shortening the timeline—hence this view looks to me like a case of plausible DSA from narrow AI with specialized abilities. Although point taken that it won’t look that narrow to most people today.
I tentatively endorse this summary. Thanks! And double thanks for the links on scaling laws.
I’m imagining doom via APS-AI that can’t necessarily control robots or do much in the physical world, but can still be very persuasive to most humans and accumulate power in the normal ways (by convincing people to do what you want, the same way every politician, activist, cult leader, CEO, general, and warlord does it). If this is classified as narrow AI, then sure, that’s a case of narrow AI takeover.
Summary of why I think the post’s estimates are too low as estimates of what’s required for a system capable of seizing a decisive strategic advantage:
Each of those problems adds on a few extra OOMs that aren’t accounted for in e.g. the setup for Omegastar (which can transfer learn across tens of thousands of games, each requiring thousands of action steps to win in a much less complicated environment than the real world).
You’d need something that can transfer learn across tens of thousands of ‘games’ each requiring billions of action steps, each one of which has way more sensory input to parse than StarCraft per time step.
When you correct Omegastar’s requirements by adding on (1) a factor for number of action steps needed to win a war Vs win a game of StarCraft, (2) a factor for the real world Vs StarCraft’s complexity of sensory input and. When you do this, the total requirement would look more like Ajeya’s reports.
I still get the intuition that OmegaStar would not just be a fancy game player! I find it hard to think about what it would be like—maybe good at gaming quite constrained systems or manipulating people?
Therefore, I think the arguments provide a strong case (unless scaling laws break—which I also think is fairly likely for technical reasons) for ‘something crazy happening by 2030’ but less strong a case for ‘AI takeover by 2030’
Summary of mine and Daniel’s disagreements:
(1) Horizon Length: Daniel thinks we’ll get a long way towards planning over a billion action steps ‘for free’ if we transfer learn over lots of games that take a thousand action steps each—so the first correction factor I gave is a lot smaller than it seems just by comparing the raw complexity of StarCraft Vs fighting a war
(2) No Robots: the complexity in sensory input difference doesn’t matter since the system won’t need to control robots [Or, as I should have also said, build robot-level models of the external world even if you’re not running the actuators yourself] - so the second correction factor isn’t an issue, because-
(3) Lower capability threshold: to take a DSA doesn’t require as many action steps as it seems or as many capabilities as often assumed. You can just do it by taking to people and over a smaller number of action steps than it would take to conquer the world yourself.
To me, it seems like Daniel’s view on horizon length reducing one of the upward corrections (1) is doing less total work than (2) and (3) in terms of shortening the timeline—hence this view looks to me like a case of plausible DSA from narrow AI with specialized abilities. Although point taken that it won’t look that narrow to most people today.
(Re scaling laws—there’s a whole debate I about how scaling laws are just a v crude observable for what’s really going on, so we shouldn’t be confident in extrapolation. This is also all conditional on the underlying assumptions of these forecasting models being correct.)
I tentatively endorse this summary. Thanks! And double thanks for the links on scaling laws.
I’m imagining doom via APS-AI that can’t necessarily control robots or do much in the physical world, but can still be very persuasive to most humans and accumulate power in the normal ways (by convincing people to do what you want, the same way every politician, activist, cult leader, CEO, general, and warlord does it). If this is classified as narrow AI, then sure, that’s a case of narrow AI takeover.