I’m curious about your derailment odds. The definition of “transformative AGI” in the paper is restrictive:
AI that can quickly and affordably be trained to perform nearly all economically and strategically valuable tasks at roughly human cost or less.
A narrow superintelligence that can, for example, engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition. I guess that would qualify as “AI-caused delay”? To follow the paper’s model, we need to estimate these odds in a conditional world where humans are not regulating AI use in ways that significantly delay the path to transformative AGI, which further increases the risk.
engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition.
I think humans could already do those things pretty well without AI, if they wanted to. Narrow AI might make those things easier, possibly much easier, just like nukes and biotech research have in the past. I agree this increases the chance that things go “off the rails”, but I think once you have an AI that can solve hard engineering problems in the real world like that, there’s just not that much further to go to full-blown superintelligence, whether you call its precursor “narrow” or not.
The probabilities in my OP are mostly just a gut sense wild guess, but they’re based on the intuition that it takes a really big derailment to halt frontier capabilities progress, which mostly happens in well-funded labs that have the resources and will to continue operating through pretty severe “turbulence”—economic depression, war, pandemics, restrictive regulation, etc. Even if new GPU manufacturing stops completely, there are already a lot of H100s and A100s lying around, and I expect that those are sufficient to get pretty far.
I’m curious about your derailment odds. The definition of “transformative AGI” in the paper is restrictive:
A narrow superintelligence that can, for example, engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition. I guess that would qualify as “AI-caused delay”? To follow the paper’s model, we need to estimate these odds in a conditional world where humans are not regulating AI use in ways that significantly delay the path to transformative AGI, which further increases the risk.
I think humans could already do those things pretty well without AI, if they wanted to. Narrow AI might make those things easier, possibly much easier, just like nukes and biotech research have in the past. I agree this increases the chance that things go “off the rails”, but I think once you have an AI that can solve hard engineering problems in the real world like that, there’s just not that much further to go to full-blown superintelligence, whether you call its precursor “narrow” or not.
The probabilities in my OP are mostly just a gut sense wild guess, but they’re based on the intuition that it takes a really big derailment to halt frontier capabilities progress, which mostly happens in well-funded labs that have the resources and will to continue operating through pretty severe “turbulence”—economic depression, war, pandemics, restrictive regulation, etc. Even if new GPU manufacturing stops completely, there are already a lot of H100s and A100s lying around, and I expect that those are sufficient to get pretty far.