This argument sounds to me like “Sparks are not wildfires. Even campfires are not wildfires. Until the fire is huge and raging out of control, unable to be easily extinguished, it is not the scary thing.”
Sure. Ok. That’s not what I mean. I mean: this small harmless thing (sparks) seems to me to have the potential to lead rapidly to the dangerous thing in the future (wildfire).
The point I’m trying to make is that the time to take preparatory action against wildfires is before they are present where you are.
What I’m saying is that I am standing deep in a drought-striken forest with no way out. I am watching some reckless children trying to build a bonfire. Currently they are managing to make sparks. The sparks haven’t caught yet, they do not self-sustain or grow. I am nevertheless predicting that soon the massive amount of optimization power being applied to this goal will show progress. The sparks will become a small fire, able to sustain if cared for and supplied with adequate material and care (e.g. shelter from extinguishment).
If you have ever built a fire with only ancient tools, you know that the step from no sparks to sparks can actually be the hardest. The next hardest step is small spark to small ember. After that things get rapidly easier and faster. The exothermic reaction gets stronger, more easily sustained, more easily increased, less picky about the quality of the fuel it is given. This pattern holds true all the way up to an out-of-control wildfire. Campfire to bonfire is easy. Bonfire or campfire to wildfire is so easy that it commonly happens by accident in dry woods, which is why campfires are forbidden even to people who claim intent to keep their fires small and under control. Too easy.
There are sparks all over the place right now, sure. What matters is which of them get to contribute to wildfires, and which are lost in those wildfires started by completely different sparks, never given the opportinity to develop because of the order in which things happen. Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI. Especially in actuality where something else happens first and ends their relevance, rather than in principle where in isolation and with enough resources they could be developed further.
Specificity is important when there are so many sparks. So I’m gesturing at a specific spark, automation of routine R&D, that plausibly might actually cause a wildfire before the other sparks grow similarly dangerous. Maybe it doesn’t catch, but in that case the other things still need to have a path towards learning of novel RLVR-level skills to have the potential, and many of them probably don’t, at least on their own.
Non-specific sparks matter for defense in depth, as in computer security where you harden the system against even the patently impossible interventions wherever it’s not too costly to do that. AI is existentially dangerous and nothing remotely close to the current methods can change that. But most sparks don’t matter for forecasting what is going to actually happen.
Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI.
Yes, that’s fair. I am claiming that these sparks are relevant. I am making a prediction. I am not proving it. I could very well be wrong. That’s an important distinction.
This argument sounds to me like “Sparks are not wildfires. Even campfires are not wildfires. Until the fire is huge and raging out of control, unable to be easily extinguished, it is not the scary thing.”
Sure. Ok. That’s not what I mean. I mean: this small harmless thing (sparks) seems to me to have the potential to lead rapidly to the dangerous thing in the future (wildfire).
The point I’m trying to make is that the time to take preparatory action against wildfires is before they are present where you are.
What I’m saying is that I am standing deep in a drought-striken forest with no way out. I am watching some reckless children trying to build a bonfire. Currently they are managing to make sparks. The sparks haven’t caught yet, they do not self-sustain or grow. I am nevertheless predicting that soon the massive amount of optimization power being applied to this goal will show progress. The sparks will become a small fire, able to sustain if cared for and supplied with adequate material and care (e.g. shelter from extinguishment).
If you have ever built a fire with only ancient tools, you know that the step from no sparks to sparks can actually be the hardest. The next hardest step is small spark to small ember. After that things get rapidly easier and faster. The exothermic reaction gets stronger, more easily sustained, more easily increased, less picky about the quality of the fuel it is given. This pattern holds true all the way up to an out-of-control wildfire. Campfire to bonfire is easy. Bonfire or campfire to wildfire is so easy that it commonly happens by accident in dry woods, which is why campfires are forbidden even to people who claim intent to keep their fires small and under control. Too easy.
There are sparks all over the place right now, sure. What matters is which of them get to contribute to wildfires, and which are lost in those wildfires started by completely different sparks, never given the opportinity to develop because of the order in which things happen. Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI. Especially in actuality where something else happens first and ends their relevance, rather than in principle where in isolation and with enough resources they could be developed further.
Specificity is important when there are so many sparks. So I’m gesturing at a specific spark, automation of routine R&D, that plausibly might actually cause a wildfire before the other sparks grow similarly dangerous. Maybe it doesn’t catch, but in that case the other things still need to have a path towards learning of novel RLVR-level skills to have the potential, and many of them probably don’t, at least on their own.
Non-specific sparks matter for defense in depth, as in computer security where you harden the system against even the patently impossible interventions wherever it’s not too costly to do that. AI is existentially dangerous and nothing remotely close to the current methods can change that. But most sparks don’t matter for forecasting what is going to actually happen.
Yes, that’s fair. I am claiming that these sparks are relevant. I am making a prediction. I am not proving it. I could very well be wrong. That’s an important distinction.