For a while, everyone was underinvesting in AI because they were not sufficiently AGI-pilled. Now, I think a lot of people (the big companies, investors, people in this community) might be overinvesting in AI because they’re not sufficiently AGI-pilled.
AI data center capex is approaching $1 trillion per year. Enough gigawatts are already planned to multiply the AI labs’ compute several times over in the next few years.
And yet the models are going crazy. I feel like the new revelations about misaligned agent swarms from OpenAI are not yet priced in. How can we possibly keep accelerating for another three years like this? If nothing else cybersecurity is surely going to break.
I can see a few possible futures:
Governments step in and regulate pre-emptively before a major catastrophe.
There is a major catastrophe and everyone is forced to slow down.
We fully lose control and AI takes over.
But another three years of just more scaling and acceleration like we’ve had the last three is increasingly infeasible. We are entering the danger zone and the race cannot just carry on as before.
I wouldn’t expect markets to price in the third possible future (total loss of control), but they should price in the first two. I think we could soon see a market correction when investors see that a deceleration will be required to keep models under control.
Economically it is probably rational for the world to be investing $1 trillion in capabilities and $10 trillion in safety. Unless you’re pessimistic, in which case we should spend $0 trillion on capabilities and still $10 trillion on safety.
This assumes that safety is the kind of good you can invest in. Right now that feels about as true as investing in peace or investing in love or similarly abstract concepts. Approximately nobody who is mainly money-bottlenecked has demonstrated a compelling track record of turning money into safety.
My main issue is that the world will find it hard to invest anything like ten trillion dollars. This is around a half of Chinese annual GDP, a quarter of the USA’s debt or a third of the USA’s annual GDP. Additionally, the world’s understanding of the AIs’ wildly transformative potential is surprisingly poor and prevents the world from making the relevant investments, especially during the looming financial crisis.
The idea is the faster safety is solved, the faster we can scale capabilities safely, which increases the growth rate of the economy from ~3% to ~100% and makes people immortal. If people want to maximize something like their discounted log(consumption) over the next 100 years, a wartime investment is warranted unless you think we couldn’t solve AI safety well enough to cure aging and automate the economy in a few decades.
I’m definitely on the pessimistic side, but yes I can see that spending more on safety may shorten the timeline for solving death which is probably a good thing if it doesn’t also increase s-risks.
My main worry is the following couple of doom scenarios.
Scenario A:
The amount of investments which was already priced in by Some Lab becomes reduced by external shocks, the lack of priced-in capabilities or misalignment;
Some Lab decides to avoid becoming bankrupt by gaining the revenue, to deliver the capabilities and uses potentially hazardous architectures like neuralese;
The capabilities emerge, but the AIs aren’t aligned.
Scenario B:
The $2,4T of investments which were already priced in by AI-2040 become reduced by external shocks, misalignment or the lack of priced-in capabilities;
Capabilities do slow down far more than AI-2040 predicted, but China keeps racing to produce compute;
The leadership is lost to China where safety is far worse, but the USA and China don’t do things like cross-audits of training runs.
Superintelligence safety isn’t “far worse” in China. You can’t get worse than zero superintelligence safety, which is what we currently have in both the US and China.
For a while, everyone was underinvesting in AI because they were not sufficiently AGI-pilled. Now, I think a lot of people (the big companies, investors, people in this community) might be overinvesting in AI because they’re not sufficiently AGI-pilled.
AI data center capex is approaching $1 trillion per year. Enough gigawatts are already planned to multiply the AI labs’ compute several times over in the next few years.
And yet the models are going crazy. I feel like the new revelations about misaligned agent swarms from OpenAI are not yet priced in. How can we possibly keep accelerating for another three years like this? If nothing else cybersecurity is surely going to break.
I can see a few possible futures:
Governments step in and regulate pre-emptively before a major catastrophe.
There is a major catastrophe and everyone is forced to slow down.
We fully lose control and AI takes over.
But another three years of just more scaling and acceleration like we’ve had the last three is increasingly infeasible. We are entering the danger zone and the race cannot just carry on as before.
I wouldn’t expect markets to price in the third possible future (total loss of control), but they should price in the first two. I think we could soon see a market correction when investors see that a deceleration will be required to keep models under control.
Economically it is probably rational for the world to be investing $1 trillion in capabilities and $10 trillion in safety. Unless you’re pessimistic, in which case we should spend $0 trillion on capabilities and still $10 trillion on safety.
This assumes that safety is the kind of good you can invest in. Right now that feels about as true as investing in peace or investing in love or similarly abstract concepts. Approximately nobody who is mainly money-bottlenecked has demonstrated a compelling track record of turning money into safety.
My main issue is that the world will find it hard to invest anything like ten trillion dollars. This is around a half of Chinese annual GDP, a quarter of the USA’s debt or a third of the USA’s annual GDP. Additionally, the world’s understanding of the AIs’ wildly transformative potential is surprisingly poor and prevents the world from making the relevant investments, especially during the looming financial crisis.
$0 trillion on capabilities and $1 trillion on safety would be equally fine.
The idea is the faster safety is solved, the faster we can scale capabilities safely, which increases the growth rate of the economy from ~3% to ~100% and makes people immortal. If people want to maximize something like their discounted log(consumption) over the next 100 years, a wartime investment is warranted unless you think we couldn’t solve AI safety well enough to cure aging and automate the economy in a few decades.
I’m definitely on the pessimistic side, but yes I can see that spending more on safety may shorten the timeline for solving death which is probably a good thing if it doesn’t also increase s-risks.
My main worry is the following couple of doom scenarios.
Scenario A:
The amount of investments which was already priced in by Some Lab becomes reduced by external shocks, the lack of priced-in capabilities or misalignment;
Some Lab decides to avoid becoming bankrupt by gaining the revenue, to deliver the capabilities and uses potentially hazardous architectures like neuralese;
The capabilities emerge, but the AIs aren’t aligned.
Scenario B:
The $2,4T of investments which were already priced in by AI-2040 become reduced by external shocks, misalignment or the lack of priced-in capabilities;
Capabilities do slow down far more than AI-2040 predicted, but China keeps racing to produce compute;
The leadership is lost to China where safety is far worse, but the USA and China don’t do things like cross-audits of training runs.
Superintelligence safety isn’t “far worse” in China. You can’t get worse than zero superintelligence safety, which is what we currently have in both the US and China.