What I’ve learned from this inspiring op-ed is that I have the ability to choose the reality I live in. I can choose to live in a reality in which AI is not a potent extinction risk just like a dinosaur living 66 million years ago could have chosen not to be dinosaur or to be a dinosaur if that is what he really wanted to be, but to choose a career path that would have insulated him from the effects of meteor strikes and other causes of extinctions.
Glad to have inspired you. Looking at your profile, your P(doom) is 92%. You can think of this op Ed as focusing on the probability space that you assign 8% to
About 5% of that 8% are worlds in which large training runs are banned soon, and all publication and discussion of algorithmic improvements to general-purpose AIs are also banned, and companies that run more than about 100 GPUs or use general-purpose AI to derive more than a few million of annual revenue or have accepted investment for any business plan in which AI is heavily used are closely monitored by some governmental agency for compliance with the aforementioned bans—giving humanity much needed time to think and come up with a decent plan for superalignment.
Let me clarify. In most of the 5% of worlds, people and companies may keep on operating the models (e.g., GPT 5.5) that have already seen widespread deployment on the theory that any truly catastrophic damage they are capable of causing would have been caused by now. (And they are useful.)
Anything that cannot outwit the FBI poses an acceptably-low level of extinction risk IMHO. No matter how instances of GPT 5.5 are arranged or combined, the combination is quite unlikely to be able to outwit the FBI in a decisive and permanent manner because of fundamental limitations in GPT 5.5 relative to the human cognitive architecture that I am very confident (but not completely certain) would require training up a new fundamental model to overcome.
Please let me know if you disagree with that last sentence.
(I use the FBI as an example of a competent and well-resourced human-controlled organization, not to suggest that the FBI will be central to stopping any AI-controlled system or process that needs to be stopped.)
I am not “outwitting the FBI” is a good operationalization, but I agree that whether through misalignment or misuse, none of the existing models (eg GPT 5.5, Opus 4.8, Myhtos, Gemini etc) can cause human extinction
What I’ve learned from this inspiring op-ed is that I have the ability to choose the reality I live in. I can choose to live in a reality in which AI is not a potent extinction risk just like a dinosaur living 66 million years ago could have chosen not to be dinosaur or to be a dinosaur if that is what he really wanted to be, but to choose a career path that would have insulated him from the effects of meteor strikes and other causes of extinctions.
Glad to have inspired you. Looking at your profile, your P(doom) is 92%. You can think of this op Ed as focusing on the probability space that you assign 8% to
About 5% of that 8% are worlds in which large training runs are banned soon, and all publication and discussion of algorithmic improvements to general-purpose AIs are also banned, and companies that run more than about 100 GPUs or use general-purpose AI to derive more than a few million of annual revenue or have accepted investment for any business plan in which AI is heavily used are closely monitored by some governmental agency for compliance with the aforementioned bans—giving humanity much needed time to think and come up with a decent plan for superalignment.
So the Op Ed is only relevant in the 3% of the probability space where people need advice for adapting economically for AI
Let me clarify. In most of the 5% of worlds, people and companies may keep on operating the models (e.g., GPT 5.5) that have already seen widespread deployment on the theory that any truly catastrophic damage they are capable of causing would have been caused by now. (And they are useful.)
Anything that cannot outwit the FBI poses an acceptably-low level of extinction risk IMHO. No matter how instances of GPT 5.5 are arranged or combined, the combination is quite unlikely to be able to outwit the FBI in a decisive and permanent manner because of fundamental limitations in GPT 5.5 relative to the human cognitive architecture that I am very confident (but not completely certain) would require training up a new fundamental model to overcome.
Please let me know if you disagree with that last sentence.
(I use the FBI as an example of a competent and well-resourced human-controlled organization, not to suggest that the FBI will be central to stopping any AI-controlled system or process that needs to be stopped.)
I am not “outwitting the FBI” is a good operationalization, but I agree that whether through misalignment or misuse, none of the existing models (eg GPT 5.5, Opus 4.8, Myhtos, Gemini etc) can cause human extinction