I am sitting here crying as the last remaining bits of diplomatic goodwill and hope for internationally coordinated treaties on coordinating the AI takeoff evaporates.
We can still hope that we won’t get AGI in the next couple of years. Society’s attitude towards AI is already negative, and we’re even seeing some congressmen openly discuss the existential risks. This growing awareness might just lead to meaningful policy changes in the future.
plausibly about 3e26 FLOPs
Or 6e26 (in FP8 FLOPs).
And already on February 17th, Colossus had 150k+ GPU. It seems that in the April message they were talking about 200k GPUs. Judging by Musk’s interview, this could mean 150,000 H100 and 50,000 H200. Perhaps the time and GPU were enough to train a GPT-5 scale model?
I sympathize with this line of thinking, but I’ve never understood something like P(doom)>0.8.
The analogies with cancer or poison seem a bit odd, because we’re trying to estimate the probability of an event that has never happened before. Without relying on anything like physical laws, without anything close to consensus. Even among the people who proposed the key ideas of the AI Risk discussions, not all were confident pessimists.
We have too many unknowns. We don’t know when superintelligence will appear. We can’t predict how governments and corporations will treat AI in the coming years. We don’t know what will happen if someone tries to use a sufficiently advanced AI for automated safety research. Or narrow AI might change the situation in the world before superintelligence appears. Our civilization could collapse for any number of reasons.
And I don’t think we can say for sure what superintelligence will do to humans.
Earlier, you wrote about a change to your AGI timelines.
What about p(doom)? It seems that in recent months there have been reasons for both optimism and pessimism.
It seems a little surprising to me how rarely confident pessimists (p(doom)>0.9) they argue with moderate optimists (p(doom)≤0.5).
I’m not specifically talking about this post. But it would be interesting if people revealed their disagreement more often.
Thanks for the reply. I remembered a recent article by Evans and thought that reasoning models might show a different behavior. Sorry if this sounds silly
Are you planning to test this on reasoning models?
I agree. But now people write so often about short timelines that it seems appropriate to recall the possible reason for the uncertainty.
Doesn’t that seem like a reason to be optimistic about reasoning models?
There doesn’t seem to be a consensus that ASI will be created in the next 5-10 years. This means that current technology leaders and their promises may be forgotten.
Does anyone else remember Ben Goertzel and Novamente? Or Hugo de Garis?
Yudkowsky may think that the plan ‘Avert all creation of superintelligence in the near and medium term — augment human intelligence’ has <5% chance of success, but your plan has <<1% chance. Obviously, you and he disagree not only on conclusions, but also on models.
It seems that we are already at the GPT 4.5 level? Except that reasoning models have confused everything, and increasing OOM on output can have the same effect as ~OOM on training, as I understand it.
By the way, you’ve analyzed the scaling of pretraining a lot. But what about inference scaling? It seems that o3 has already used thousands of GPUs to solve tasks in ARC-AGI.
Thank you. In conditions of extreme uncertainty about the timing and impact of AGI, it’s nice to know at least something definite.
Can we assume that Gemini 2.0, GPT-4o, Claude 3.5 and other models with similar performance have a similar compute?
If we don’t build fast enough, then the authoritarian countries could win.
Ideally it would be something like the UN, but given the geopolitical complexities, that doesn’t seem very possible.
This sounds like a rejection of international coordination.
But there was coordination between the United States and the USSR on nuclear weapons issues, despite geopolitical tensions, for example. You can interact with countries you don’t like without trying to destroy the world faster than them!
2 years ago, you seemed quite optimistic about AGI Safety/Alignment and had a long timeline.
Have your views changed since then?
I understand that hiring will be necessary in any case.
Keeping people as a commodity for acasual trade or pets seems like a more likely option.
If only one innovation separates us from AGI, we’re fucked.
It seems that if OpenAI or Anthropic had agreed with you, they should have had even shorter timelines.
Is this the result you expect by default? Or is this just one of many unlikely scenarios (like Hanson’s ‘The Age of Em’) that are worth considering?