I feel pretty worried that this is the calm before the storm. I think starting with late 2026, we are officially in something like “crunch time.”
I concretely mean that it seems like things are really speeding up and the stakes are getting higher.
We have to treat our multiple-month research projects in the mindset of “this is likely one of the last research projects I ever do before this kind of work is automated by AIs.”
The kinds of engagements, setups, and precedents that are set by labs and third parties right now will substantially dictate the shape of takeoff. The time to pilot whatever things we want during takeoff is right now. We can’t do it later. If we want to do major pacing at any point, we need to start laying the foundations right now. We just don’t have much time to get that many reps in.
Relatedly: ‘I think pausing/pacing the frontier is less of “an action that you can only do once and thus have to conserve until dangerously late into AI takeoff” and more of a hazy set of actions, all of which can be practiced, and most of which get easier if you have earlier precedent for them. It seems really productive to set up systems right now where labs can take net-positive actions and verify them to each other and the rest of the world. We can get practice with the small versions before we can achieve the large versions.’
As more high-profile incidents happen (and possibly, as the first casualties mount), there will be wild and sudden waves of pressure, both internal and external to the labs. These waves could be extremely counterproductive or extremely productive based on what sorts of asks and changes they materialize.
There will be so much more politics everywhere. Politics in the naive sense, and politics in the “balancing fragile and high-stakes relationships rather than object-level things.”
Capabilities will just. Keep. Advancing. The models will be able to do more and more things. Various thresholds I have personally set for “things get crazy when this happens” will happen.
There will plausibly be days or weeks during which history will be decided by a small number of people, primarily in the area of finding ways for the main US labs to pace AI development, and finding ways for the rest of the world to join them. A few internal and external champions of the “take things slowly and attempt to coordinate” view will likely have immense sway on human history. A few internal and external champions of “here’s a concrete proposal for how to uphold pacing” will likely also have immense sway on human history.
It will get easier to convince people that AI safety is important and ASI is a big deal. The level of attention paid to these issues will skyrocket. But also, with attention comes polarization, and with attention come distractions and counterproductive culture wars.
I really long for a day when we enter a sane global agreement for shaping AI development and we can all get together and celebrate. I think this is a plausible (>20%) future and one that’s really tractable to make more likely.
But it’s likely (>50%) that the companies will race ahead and kick off the RSI loop within three years. This means that sometime soon, we will likely enter a state where multiple companies have similarly-capable drop-in researchers/engineers, and are kicking off a crazy runaway process and trying to shape it in a way that doesn’t result in catastrophe.
Today I told myself that we’re officially in the end times now.
I don’t personally know what to do. Doesn’t feel like I have any actions available to me that will help. An avalanche is coming. Maybe I could warn a few people.
Good luck to those of you in a position to do something. I hope you can avert disaster, and I hope the utopia you steer us toward is more like Star Trek and less like some utilitarian wireheading generative-content ultra-tiktok-iverse where the light cone is tiled with iPad babies.
One thing to consider: you live in a democracy (I’m guessing), and that democracy says that a citizen’s responsibility is to talk to their fellow citizens and their representatives. It doesn’t say that your responsibility is to solve the problem on behalf of your community; that’s more than we’ve ever asked from any ordinary citizen. If you write a letter to your newspaper editor and call your House Rep and Senators and state-level reps too, you’ve done your part, and we should be thankful for it.
It will get easier to convince people that AI safety is important and ASI is a big deal. The level of attention paid to these issues will skyrocket
Interestingly, I go to a book club in my city which is mostly women in their 20s with a few people in their 40s/50s. We generally read fairly middlebrow fiction (e.g. booker prize nominees) with the occasional quite topical non-fiction (think stuff like Bad Blood). All the people there are nice liberals; AI is almost never mentioned and it’s usually in a generally negative context, like complaining about water usage or it making people stupid.
Anyway, I didn’t go to the last meeting but they upload the voting results for the next book to read and I was absolutely stunned to see that IABIED nearly won. These are exactly the kind of people I thought would dismiss x-risk as a marketing tactic.
I feel pretty worried that this is the calm before the storm. I think starting with late 2026, we are officially in something like “crunch time.”
I concretely mean that it seems like things are really speeding up and the stakes are getting higher.
We have to treat our multiple-month research projects in the mindset of “this is likely one of the last research projects I ever do before this kind of work is automated by AIs.”
The kinds of engagements, setups, and precedents that are set by labs and third parties right now will substantially dictate the shape of takeoff. The time to pilot whatever things we want during takeoff is right now. We can’t do it later. If we want to do major pacing at any point, we need to start laying the foundations right now. We just don’t have much time to get that many reps in.
Relatedly: ‘I think pausing/pacing the frontier is less of “an action that you can only do once and thus have to conserve until dangerously late into AI takeoff” and more of a hazy set of actions, all of which can be practiced, and most of which get easier if you have earlier precedent for them. It seems really productive to set up systems right now where labs can take net-positive actions and verify them to each other and the rest of the world. We can get practice with the small versions before we can achieve the large versions.’
As more high-profile incidents happen (and possibly, as the first casualties mount), there will be wild and sudden waves of pressure, both internal and external to the labs. These waves could be extremely counterproductive or extremely productive based on what sorts of asks and changes they materialize.
There will be so much more politics everywhere. Politics in the naive sense, and politics in the “balancing fragile and high-stakes relationships rather than object-level things.”
Capabilities will just. Keep. Advancing. The models will be able to do more and more things. Various thresholds I have personally set for “things get crazy when this happens” will happen.
There will plausibly be days or weeks during which history will be decided by a small number of people, primarily in the area of finding ways for the main US labs to pace AI development, and finding ways for the rest of the world to join them. A few internal and external champions of the “take things slowly and attempt to coordinate” view will likely have immense sway on human history. A few internal and external champions of “here’s a concrete proposal for how to uphold pacing” will likely also have immense sway on human history.
It will get easier to convince people that AI safety is important and ASI is a big deal. The level of attention paid to these issues will skyrocket. But also, with attention comes polarization, and with attention come distractions and counterproductive culture wars.
I really long for a day when we enter a sane global agreement for shaping AI development and we can all get together and celebrate. I think this is a plausible (>20%) future and one that’s really tractable to make more likely.
But it’s likely (>50%) that the companies will race ahead and kick off the RSI loop within three years. This means that sometime soon, we will likely enter a state where multiple companies have similarly-capable drop-in researchers/engineers, and are kicking off a crazy runaway process and trying to shape it in a way that doesn’t result in catastrophe.
Today I told myself that we’re officially in the end times now.
I don’t personally know what to do. Doesn’t feel like I have any actions available to me that will help. An avalanche is coming. Maybe I could warn a few people.
Good luck to those of you in a position to do something. I hope you can avert disaster, and I hope the utopia you steer us toward is more like Star Trek and less like some utilitarian wireheading generative-content ultra-tiktok-iverse where the light cone is tiled with iPad babies.
One thing to consider: you live in a democracy (I’m guessing), and that democracy says that a citizen’s responsibility is to talk to their fellow citizens and their representatives. It doesn’t say that your responsibility is to solve the problem on behalf of your community; that’s more than we’ve ever asked from any ordinary citizen. If you write a letter to your newspaper editor and call your House Rep and Senators and state-level reps too, you’ve done your part, and we should be thankful for it.
Interestingly, I go to a book club in my city which is mostly women in their 20s with a few people in their 40s/50s. We generally read fairly middlebrow fiction (e.g. booker prize nominees) with the occasional quite topical non-fiction (think stuff like Bad Blood). All the people there are nice liberals; AI is almost never mentioned and it’s usually in a generally negative context, like complaining about water usage or it making people stupid.
Anyway, I didn’t go to the last meeting but they upload the voting results for the next book to read and I was absolutely stunned to see that IABIED nearly won. These are exactly the kind of people I thought would dismiss x-risk as a marketing tactic.