are so much less sample efficient than the self-distillation method
For actually novel capabilities, I have severe doubt about this. Do you think OPSD could be set up to train an AlphaGo level game player? It seems unlikely to me
I don’t think that is a good bar to set. AlphaGo works so well because it is directly running an algorithm to play Go. With AlphaGo, the neural network directly outputs scores and policies for Go positions. However, auto-regressive LLMs don’t do this. They can learn to play Go, but they do this not by directly running a Go-winning algorithm, but by simulating a Go-winning algorithm with their stream of thoughts. I don’t think an auto-regressive LLM would do nearly as well as AlphaGo, simply because this simulation step is inefficient. It’s the same reason that a human can’t add 1,000,000 numbers together in a second, even though we’ve had computers that could do that for 50 years. The human brain isn’t designed for adding lots of numbers together.
The biggest novel capability I see from OPSD is that it enables language models to learn from natural language feedback, which is 99% of how humans learn intellectual skills. The fact that when humans go to school, they are taught using words rather than whips and sugar cubes, makes me believe that this is a very important capability.
I don’t disagree with the proposal being useful and likely already in use. I disagree that it can replace RL or is even in the same ballpark of sample efficiency. You said before that now anyone can train; disagree, because RL can train on novel problems and find ways to solve them, OPSD can’t if I understand correctly.
AlphaGo strikes me as much closer to OPSD than RLVR—MCTS can be seen as composing the target using the value network (often even same weights except head!), where the hint is the few moves we simulated, which we then try to strap to the policy.
I’m surprised you don’t think this could result in novel capabilities, can you explain more why?
For actually novel capabilities, I have severe doubt about this. Do you think OPSD could be set up to train an AlphaGo level game player? It seems unlikely to me
I don’t think that is a good bar to set. AlphaGo works so well because it is directly running an algorithm to play Go. With AlphaGo, the neural network directly outputs scores and policies for Go positions. However, auto-regressive LLMs don’t do this. They can learn to play Go, but they do this not by directly running a Go-winning algorithm, but by simulating a Go-winning algorithm with their stream of thoughts. I don’t think an auto-regressive LLM would do nearly as well as AlphaGo, simply because this simulation step is inefficient. It’s the same reason that a human can’t add 1,000,000 numbers together in a second, even though we’ve had computers that could do that for 50 years. The human brain isn’t designed for adding lots of numbers together.
The biggest novel capability I see from OPSD is that it enables language models to learn from natural language feedback, which is 99% of how humans learn intellectual skills. The fact that when humans go to school, they are taught using words rather than whips and sugar cubes, makes me believe that this is a very important capability.
I don’t disagree with the proposal being useful and likely already in use. I disagree that it can replace RL or is even in the same ballpark of sample efficiency. You said before that now anyone can train; disagree, because RL can train on novel problems and find ways to solve them, OPSD can’t if I understand correctly.
AlphaGo strikes me as much closer to OPSD than RLVR—MCTS can be seen as composing the target using the value network (often even same weights except head!), where the hint is the few moves we simulated, which we then try to strap to the policy.
I’m surprised you don’t think this could result in novel capabilities, can you explain more why?