Aside from whether the claims you are citing are true or well-calibrated, I want to point something out.
I actually think that your example scenarios here illustrate exactly the type of scenario that I wanted to disarm with this post. Or if not disarm them, at least give people the tools they need to disarm them.
It is very possible one lab/country gains a decisive advantage before any others. The approach to ASI Is likely to be chaotic and fraught with disagreement. It very well may not be obvious what is happening to other powers until a decisive advantage is gained. If the lab/country with a decisive advantage succeeds at technical alignment you may end up in a world which bypasses many of these concerns.
This very specific scenario, that I think could technically happen but is extremely unlikely, falls to the third filter, “nightmare singleton”: the winning AI company having to create a singleton in order to actually bypass the first and second filters.
You may believe this can be a good outcome, but you are still fundamentally trusting that the singleton established by a private company will be good for you / most people, which I don’t qualify as a satisfying solution to the third filter.
(The reason I think this scenario is unlikely is that for this to work the AI company would’ve had to get a really, really big advantage over anyone else: how did they do this without cutting too many corners on safety and failing at the second filter? And of course, the thing about the government taking over the project.)
It took the Soviets years of espionage to steal atomic secrets. If a lab is approaching ASI, one can expect that the pre-ASI AI will be heavily woven into their security architecture for securing model weights which may effectively prevent nation states from stealing them.
I could spend time arguing why I think this scenario is unlikely, but I think this would miss the main point of the post: it doesn’t address the fact that the winning AI company still needs to make a singleton, and we’re failing at the third filter.
Aside from whether the claims you are citing are true or well-calibrated, I want to point something out.
I actually think that your example scenarios here illustrate exactly the type of scenario that I wanted to disarm with this post. Or if not disarm them, at least give people the tools they need to disarm them.
This very specific scenario, that I think could technically happen but is extremely unlikely, falls to the third filter, “nightmare singleton”: the winning AI company having to create a singleton in order to actually bypass the first and second filters.
You may believe this can be a good outcome, but you are still fundamentally trusting that the singleton established by a private company will be good for you / most people, which I don’t qualify as a satisfying solution to the third filter.
(The reason I think this scenario is unlikely is that for this to work the AI company would’ve had to get a really, really big advantage over anyone else: how did they do this without cutting too many corners on safety and failing at the second filter? And of course, the thing about the government taking over the project.)
I could spend time arguing why I think this scenario is unlikely, but I think this would miss the main point of the post: it doesn’t address the fact that the winning AI company still needs to make a singleton, and we’re failing at the third filter.