Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
The detailed analysis is contained in the covert project supplement, but I can give a simple argument here. The intuition for why they go so slowly is that compute is a key driver of progress, and a covert project will have much much less compute than a typical project: in the branch we assume they have ~500k H100e, whereas there are ~200M H100e at SOY 2029 in this scenario (of which ~half are going to AI R&D).
Then yeah, there’s a question of how much algorithmic progress leaks to them; minimizing this is one of the main reasons that we try to scale via compute as opposed to via algorithmic advances in Plan A.
>(which also means that, post-pause, we’ve burned our entire lead, right?)
This isn’t true because by 2040 in the scenario we are very confident there’s no covert project because of improved technology such as lie detectors and privacy preserving AI verification. (Also, even without that, it seems very likely that a covert project of that size would be detected, but we’re less sure). Or in other words, it’s only true if you assume that we can’t detect covert projects after they are started, only right at the beginning when they are diverting their chips.
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?
Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
Thanks for the comments!
The detailed analysis is contained in the covert project supplement, but I can give a simple argument here. The intuition for why they go so slowly is that compute is a key driver of progress, and a covert project will have much much less compute than a typical project: in the branch we assume they have ~500k H100e, whereas there are ~200M H100e at SOY 2029 in this scenario (of which ~half are going to AI R&D).
Then yeah, there’s a question of how much algorithmic progress leaks to them; minimizing this is one of the main reasons that we try to scale via compute as opposed to via algorithmic advances in Plan A.
>(which also means that, post-pause, we’ve burned our entire lead, right?)
This isn’t true because by 2040 in the scenario we are very confident there’s no covert project because of improved technology such as lie detectors and privacy preserving AI verification. (Also, even without that, it seems very likely that a covert project of that size would be detected, but we’re less sure). Or in other words, it’s only true if you assume that we can’t detect covert projects after they are started, only right at the beginning when they are diverting their chips.
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?