As an example of what RSI might be like, I find it helpful to go back to OpenAI’s Dota 2 result from 2017:
This slide from Ilya’s lecture shows the bot’s Trueskill rating[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.
Note that this was using self-play (the model training itself through some feedback loop which generates its own training data), which is arguably a weaker form of RSI than classical RSI in the form of automation of AI research (the model researches new ML algorithms, like optimizers/architectures/objective functions, which are then used to train an improved successor model).
The methods don’t exclude each other, but self-play is easier to achieve and tends to plateau earlier (though possibly at superhuman levels), since it is usually limited by a suboptimal fixed ML algorithm. In contrast, automatic ML research could, in principle, scale to technological maturity, i.e., to a physically optimal ASI.
Self-play is already studied for LLM reinforcement learning, see e.g. this or this.
Scaling creates visible progress without a need for novel methods, and it’s constrained by what the available/economical compute can do with the current methods. Self-play is a way to keep scaling going where you wouldn’t otherwise have enough data.
RSI in the sense of automated R&D doesn’t necessarily imply fast progress if it can’t invent novel methods quickly, methods that make a better use of available compute, unlock scaling of something important to more of the available compute than was previously possible, or generate data that was previously in short supply or at a low quality. This could take significant time if the learning loop for deep skills is too long, longer than it is for humans. RSI is additionally less likely to imply fast progress if it starts with AIs that are already scaled beyond all reason and are still stumbling unevenly around human level. Even so, this could be centrally RSI, fitting the intended sense of the term. The AIs like that are perhaps even capable of inventing superintelligence eventually, but it could take a while.
As an example of what RSI might be like, I find it helpful to go back to OpenAI’s Dota 2 result from 2017:
This slide from Ilya’s lecture shows the bot’s Trueskill rating[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.
Similar to Elo in Chess and other games
Note that this was using self-play (the model training itself through some feedback loop which generates its own training data), which is arguably a weaker form of RSI than classical RSI in the form of automation of AI research (the model researches new ML algorithms, like optimizers/architectures/objective functions, which are then used to train an improved successor model).
The methods don’t exclude each other, but self-play is easier to achieve and tends to plateau earlier (though possibly at superhuman levels), since it is usually limited by a suboptimal fixed ML algorithm. In contrast, automatic ML research could, in principle, scale to technological maturity, i.e., to a physically optimal ASI.
Self-play is already studied for LLM reinforcement learning, see e.g. this or this.
Scaling creates visible progress without a need for novel methods, and it’s constrained by what the available/economical compute can do with the current methods. Self-play is a way to keep scaling going where you wouldn’t otherwise have enough data.
RSI in the sense of automated R&D doesn’t necessarily imply fast progress if it can’t invent novel methods quickly, methods that make a better use of available compute, unlock scaling of something important to more of the available compute than was previously possible, or generate data that was previously in short supply or at a low quality. This could take significant time if the learning loop for deep skills is too long, longer than it is for humans. RSI is additionally less likely to imply fast progress if it starts with AIs that are already scaled beyond all reason and are still stumbling unevenly around human level. Even so, this could be centrally RSI, fitting the intended sense of the term. The AIs like that are perhaps even capable of inventing superintelligence eventually, but it could take a while.