First, you train AlphaGo on expert human examples. This is enough to beat Lee Sedol and Ke Jie. Then, you train AlphaZero purely through self-play. It destroys AlphaGo after only a few hours.
First, you train RL agents on human playthroughs of Minecraft. They do okay. Then, DreamerV3 learns entirely by itself and becomes the first to get diamonds.
First, you train theorem provers on human proofs. Then, you train AlphaProof using AlphaZero and you get silver on IMO for the first time.
First, you pretrain a language model on all human data. Then...
This feels like a special case of the bitter lesson, but it’s not the same thing. It seems to rely on the distinction between prediction and search latent in ideas like AISI. It’s the kind of thing that I’m sure Gwern has christened in some comment lost to the internet’s backwaters. We should have a name for it—something more refined than just “foom.”
I think this is important because the safety community still isn’t thinking very much about search & RL, even after all the recent progress with reasoning models. We’ve updated very far away from AlphaZero as a reference class, and I think we will regret this.
On the other hand, the ideas I’m talking about here seem to have widespread recognition among people working on capabilities. Demis is very transparent about where they’re headed with language models, AlphaZero, and open-ended exploration (e.g., at 20:48). Noam Brown is adamant about test-time scaling/reasoning being the future (e.g., at 20:32). I think R1 has driven the message home for everyone else.
To be fair here, AlphaZero was a case where it not only had an essentially unhackable reward model, but also could generate very large amounts of data, which while not totally unique to Go or gaming, is a property that is generally hard to come by in a lot of domains, so progress will probably be slower than AlphaZero.
Also, a lot of the domains are areas where latencies are either very low or you can tolerate long latency, which is not the case in the physical world very often.
I don’t think I get what phenomenon you’re pointing to.
Your first bullet point makes it sound like AlphaGo wasn’t trained using self-play, in contrast to AlphaZero. However, AlphaGo was trained with a combination of supervised learning and self-play. They removed the supervised learning part from AlphaZero to make it simpler and more general.
DreamerV3 also fits the pattern where previous SOTA approaches used a combination of imitation learning and reinforcement learning, while DreamerV3 was able to remove the imitation learning part.[1]
To my understanding, AlphaProof was trained by translating a bunch of math problems to Lean, and using “correct proof” as reward for AlphaZero. This approach also combines human data (our math problems) with reinforcement learning (AlphaZero).
Your final example feels close to AlphaProof if you finish it with “Then you finetune CoT with reinforcement learning to yield impressive performance on reasoning benchmarks”, but I don’t think that’s what you were going for.
The first two examples seem covered by “when reinforcement learning works well, imitation learning is no longer needed”. Idk about the rest.
Could you clarify by giving more examples or otherwise explain what you’re looking for?
I got curious how DreamerV3 figures out Minecraft with nothing to imitate and no intermediate reward, so I checked the paper. There are intermediate rewards. They give +1 reward for each of 12 ordered milestones leading up to the diamond, and −0.01 for each lost heart and +0.01 for each restored heart. Additionally, they use “the block breaking setting of prior work[19] because the provided action space would make it challenging for stochastic policies to keep a key pressed for a prolonged time”. So to get started, probably the agent manages to randomly break a tree block and get its first reward.
With AlphaProof, the relevant piece is that the solver network generates its own proofs and disproofs to train against. There’s no imitation learning after formalization. There is a slight disanalogy where, for formalization, we mostly jumped straight to self-play/search, and I don’t think there was ever a major imitation-learning-based approach (though I did find at least one example).
Your quote “when reinforcement learning works well, imitation learning is no longer needed” is pretty close to what I mean. What I’m actually trying to get at is a stronger statement: we often bootstrap using imitation learning to figure out how to get the reinforcement learning component working initially, but once we do, we can usually discard the imitation learning entirely.
do we have good reason to think they didn’t specifically train it on human lean proofs? it seems plausible to me that they did but idk
the curriculum of human problems teaches it human tricks
lean sorta “knows” a bunch of human tricks
We could argue about whether AlphaProof “is mostly human imitation or mostly RL”, but I feel like it’s pretty clear that it’s more analogous to AlphaGo than to AlphaZero.
Because if a planner is going to be the best, it needs to be capable of finding unusual (better!) plans. If it’s capable of finding those, there’s ~no benefit of knowing the conventional wisdom about how to do it (climbing slang: beta).
Nitpick: first alphago was trained by a combination of supervised learning from human expert games and reinforcement learning from self-play. Also, Ke Jie was beaten by AlphaGo Master which was a version at a later stage of development.
Yes, my original comment wasn’t clear about this, but your nitpick is actually a key part of what I’m trying to get at.
Usually, you start with imitation learning and tack on RL at the end. That’s what AlphaGo is. It’s what predecessors to Dreamer-V3 like VPT are. It’s what current reasoning models are.
But then, eventually, you figure out how to bypass the imitation learning/behavioral cloning part and do RL from the start. Human priors serve as a temporary bootstrapping mechanism until we develop approaches that can learn effectively from scratch.
>Human priors serve as a temporary bootstrapping mechanism until we develop approaches that can learn effectively from scratch.
I would argue that instead human priors serve as a mechanism to help the search process, as it’s being shown with cold-started reasoning models: they bake-in some reasoning traces that the model can then learn to exploit via RL. While this is not very bitter lesson-esque, the solution space is so large that it’d probably be quite difficult to do so without the cold start phase (although R1-zero kind of hints at this being possible). Maybe we have not yet thrown as much compute at the problem to do this search from scratch effectively.
The Bitter Lesson is pretty on point but you could call it “Bootstrapping from Zero”, the “Autodidactic Leap”, the “Self-Discovery Transition”, or “Breaking the Imitation Ceiling” if you prefer.
DreamerV3 is not a great example, as they use so many hacks to make the task easier that it barely counts as getting a diamond or Minecraft anymore. Action shaping, macro actions, instant block breaking, fake “bug fixing”, all to get a diamond in 0.4% of episodes.
What do you call this phenomenon?
First, you train AlphaGo on expert human examples. This is enough to beat Lee Sedol and Ke Jie. Then, you train AlphaZero purely through self-play. It destroys AlphaGo after only a few hours.
First, you train RL agents on human playthroughs of Minecraft. They do okay. Then, DreamerV3 learns entirely by itself and becomes the first to get diamonds.
First, you train theorem provers on human proofs. Then, you train AlphaProof using AlphaZero and you get silver on IMO for the first time.
First, you pretrain a language model on all human data. Then...
This feels like a special case of the bitter lesson, but it’s not the same thing. It seems to rely on the distinction between prediction and search latent in ideas like AISI. It’s the kind of thing that I’m sure Gwern has christened in some comment lost to the internet’s backwaters. We should have a name for it—something more refined than just “foom.”
I think this is important because the safety community still isn’t thinking very much about search & RL, even after all the recent progress with reasoning models. We’ve updated very far away from AlphaZero as a reference class, and I think we will regret this.
On the other hand, the ideas I’m talking about here seem to have widespread recognition among people working on capabilities. Demis is very transparent about where they’re headed with language models, AlphaZero, and open-ended exploration (e.g., at 20:48). Noam Brown is adamant about test-time scaling/reasoning being the future (e.g., at 20:32). I think R1 has driven the message home for everyone else.
To be fair here, AlphaZero was a case where it not only had an essentially unhackable reward model, but also could generate very large amounts of data, which while not totally unique to Go or gaming, is a property that is generally hard to come by in a lot of domains, so progress will probably be slower than AlphaZero.
Also, a lot of the domains are areas where latencies are either very low or you can tolerate long latency, which is not the case in the physical world very often.
We already have seen a lot of progress in this regard with the new reasoning models, see this neglected post for details.
I don’t think I get what phenomenon you’re pointing to.
Your first bullet point makes it sound like AlphaGo wasn’t trained using self-play, in contrast to AlphaZero. However, AlphaGo was trained with a combination of supervised learning and self-play. They removed the supervised learning part from AlphaZero to make it simpler and more general.
DreamerV3 also fits the pattern where previous SOTA approaches used a combination of imitation learning and reinforcement learning, while DreamerV3 was able to remove the imitation learning part.[1]
To my understanding, AlphaProof was trained by translating a bunch of math problems to Lean, and using “correct proof” as reward for AlphaZero. This approach also combines human data (our math problems) with reinforcement learning (AlphaZero).
Your final example feels close to AlphaProof if you finish it with “Then you finetune CoT with reinforcement learning to yield impressive performance on reasoning benchmarks”, but I don’t think that’s what you were going for.
The first two examples seem covered by “when reinforcement learning works well, imitation learning is no longer needed”. Idk about the rest.
Could you clarify by giving more examples or otherwise explain what you’re looking for?
I got curious how DreamerV3 figures out Minecraft with nothing to imitate and no intermediate reward, so I checked the paper. There are intermediate rewards. They give +1 reward for each of 12 ordered milestones leading up to the diamond, and −0.01 for each lost heart and +0.01 for each restored heart. Additionally, they use “the block breaking setting of prior work[19] because the provided action space would make it challenging for stochastic policies to keep a key pressed for a prolonged time”. So to get started, probably the agent manages to randomly break a tree block and get its first reward.
With AlphaProof, the relevant piece is that the solver network generates its own proofs and disproofs to train against. There’s no imitation learning after formalization. There is a slight disanalogy where, for formalization, we mostly jumped straight to self-play/search, and I don’t think there was ever a major imitation-learning-based approach (though I did find at least one example).
Your quote “when reinforcement learning works well, imitation learning is no longer needed” is pretty close to what I mean. What I’m actually trying to get at is a stronger statement: we often bootstrap using imitation learning to figure out how to get the reinforcement learning component working initially, but once we do, we can usually discard the imitation learning entirely.
I think AlphaProof is pretty far from being just RL from scratch:
they use a pretrained language model; I think the model is trained on human math in particular ( https://archive.is/Cwngq#selection-1257.0-1272.0:~:text=Dr. Hubert’s team,frequency was reduced. )
do we have good reason to think they didn’t specifically train it on human lean proofs? it seems plausible to me that they did but idk
the curriculum of human problems teaches it human tricks
lean sorta “knows” a bunch of human tricks
We could argue about whether AlphaProof “is mostly human imitation or mostly RL”, but I feel like it’s pretty clear that it’s more analogous to AlphaGo than to AlphaZero.
(a relevant thread: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce?commentId=ZKuABGnKf7v35F5gp )
Okay, great, then we just have to wait a year for AlphaProofZero to get a perfect score on the IMO.
I propose: the best planners must break the beta.
Because if a planner is going to be the best, it needs to be capable of finding unusual (better!) plans. If it’s capable of finding those, there’s ~no benefit of knowing the conventional wisdom about how to do it (climbing slang: beta).
Edit: or maybe: good planners don’t need beta?
That’s fun but a little long. Why not… BetaZero?
Nitpick: first alphago was trained by a combination of supervised learning from human expert games and reinforcement learning from self-play. Also, Ke Jie was beaten by AlphaGo Master which was a version at a later stage of development.
Yes, my original comment wasn’t clear about this, but your nitpick is actually a key part of what I’m trying to get at.
Usually, you start with imitation learning and tack on RL at the end. That’s what AlphaGo is. It’s what predecessors to Dreamer-V3 like VPT are. It’s what current reasoning models are.
But then, eventually, you figure out how to bypass the imitation learning/behavioral cloning part and do RL from the start. Human priors serve as a temporary bootstrapping mechanism until we develop approaches that can learn effectively from scratch.
>Human priors serve as a temporary bootstrapping mechanism until we develop approaches that can learn effectively from scratch.
I would argue that instead human priors serve as a mechanism to help the search process, as it’s being shown with cold-started reasoning models: they bake-in some reasoning traces that the model can then learn to exploit via RL. While this is not very bitter lesson-esque, the solution space is so large that it’d probably be quite difficult to do so without the cold start phase (although R1-zero kind of hints at this being possible). Maybe we have not yet thrown as much compute at the problem to do this search from scratch effectively.
The Bitter Lesson is pretty on point but you could call it “Bootstrapping from Zero”, the “Autodidactic Leap”, the “Self-Discovery Transition”, or “Breaking the Imitation Ceiling” if you prefer.
DreamerV3 is not a great example, as they use so many hacks to make the task easier that it barely counts as getting a diamond or Minecraft anymore. Action shaping, macro actions, instant block breaking, fake “bug fixing”, all to get a diamond in 0.4% of episodes.
More info here: https://x.com/Karolis_Ram/status/1785750372394348632