There are sparks all over the place right now, sure. What matters is which of them get to contribute to wildfires, and which are lost in those wildfires started by completely different sparks, never given the opportinity to develop because of the order in which things happen. Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI. Especially in actuality where something else happens first and ends their relevance, rather than in principle where in isolation and with enough resources they could be developed further.
Specificity is important when there are so many sparks. So I’m gesturing at a specific spark, automation of routine R&D, that plausibly might actually cause a wildfire before the other sparks grow similarly dangerous. Maybe it doesn’t catch, but in that case the other things still need to have a path towards learning of novel RLVR-level skills to have the potential, and many of them probably don’t, at least on their own.
Non-specific sparks matter for defense in depth, as in computer security where you harden the system against even the patently impossible interventions wherever it’s not too costly to do that. AI is existentially dangerous and nothing remotely close to the current methods can change that. But most sparks don’t matter for forecasting what is going to actually happen.
Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI.
Yes, that’s fair. I am claiming that these sparks are relevant. I am making a prediction. I am not proving it. I could very well be wrong. That’s an important distinction.
There are sparks all over the place right now, sure. What matters is which of them get to contribute to wildfires, and which are lost in those wildfires started by completely different sparks, never given the opportinity to develop because of the order in which things happen. Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI. Especially in actuality where something else happens first and ends their relevance, rather than in principle where in isolation and with enough resources they could be developed further.
Specificity is important when there are so many sparks. So I’m gesturing at a specific spark, automation of routine R&D, that plausibly might actually cause a wildfire before the other sparks grow similarly dangerous. Maybe it doesn’t catch, but in that case the other things still need to have a path towards learning of novel RLVR-level skills to have the potential, and many of them probably don’t, at least on their own.
Non-specific sparks matter for defense in depth, as in computer security where you harden the system against even the patently impossible interventions wherever it’s not too costly to do that. AI is existentially dangerous and nothing remotely close to the current methods can change that. But most sparks don’t matter for forecasting what is going to actually happen.
Yes, that’s fair. I am claiming that these sparks are relevant. I am making a prediction. I am not proving it. I could very well be wrong. That’s an important distinction.