Yep, one hope is “EAs are good at forecasting how things will go and that maintains our influence. Both because we have a reputation as good forecasters so people listen to us, and because we make good decisions based on those forecasts (i.e. a similar arbitrage play as the “AI will be a bigger deal than people think, let’s pile into it”)”.
But I’m worried that [the top few] EAs are approaching the Horizon Point beyond which their impressive forecasting breaks down. Like, 10 years ago, Carl Schulman and Paul Christiano [and people in that reference class] could see in such clarity how things would go in 10 years time. But today, they can’t see another 10 years into the future. Like, maybe a few years into the future? Even there I’m sceptical. They might have run out of “alpha”.
For a concrete example, Daniel’s “what 2026 looks like” was pretty on-the-nose. But AI 2027 isn’t nearly so impressive — the distributions are so much broader, and Daniel himself is like “here’s a bunch of ways things could go.” AI 2026 didn’t have multiple scenarios!
One problem here is that our alpha was predicting the change in the technical landscape. But it’s not just technical landscape that will change. It‘s also the political landscape, which has been dormant for most of 2019-2026. And soon the geopolitical landscape will start rumbling. And the economic. And the cultural. I think these things are still pretty dormant, compared to how much they will start flipping the overall strategic landscape. (See here.)
IIUC, this is why Daniel’s current forecasts are choose-your-adventure, compared with his 2021 forecasts about 2021-2026.
Like, do we have good forecasts now which weren’t ~ “priced in” 5 years ago among Carl/Paul/Kokotajlo/etc? I feel like the uncertainties they had 5 years ago are pretty much still unresolved.
Yep, one hope is “EAs are good at forecasting how things will go and that maintains our influence. Both because we have a reputation as good forecasters so people listen to us, and because we make good decisions based on those forecasts (i.e. a similar arbitrage play as the “AI will be a bigger deal than people think, let’s pile into it”)”.
But I’m worried that [the top few] EAs are approaching the Horizon Point beyond which their impressive forecasting breaks down. Like, 10 years ago, Carl Schulman and Paul Christiano [and people in that reference class] could see in such clarity how things would go in 10 years time. But today, they can’t see another 10 years into the future. Like, maybe a few years into the future? Even there I’m sceptical. They might have run out of “alpha”.
For a concrete example, Daniel’s “what 2026 looks like” was pretty on-the-nose. But AI 2027 isn’t nearly so impressive — the distributions are so much broader, and Daniel himself is like “here’s a bunch of ways things could go.” AI 2026 didn’t have multiple scenarios!
One problem here is that our alpha was predicting the change in the technical landscape. But it’s not just technical landscape that will change. It‘s also the political landscape, which has been dormant for most of 2019-2026. And soon the geopolitical landscape will start rumbling. And the economic. And the cultural. I think these things are still pretty dormant, compared to how much they will start flipping the overall strategic landscape. (See here.)
IIUC, this is why Daniel’s current forecasts are choose-your-adventure, compared with his 2021 forecasts about 2021-2026.
Like, do we have good forecasts now which weren’t ~ “priced in” 5 years ago among Carl/Paul/Kokotajlo/etc? I feel like the uncertainties they had 5 years ago are pretty much still unresolved.
Did Carlsmith make impressive predictions? Or are you refering to Carl Shulman?
Yeah, I meant Carl Shulman. Well-spotted, thanks.