The first one (bootstrapping) has the issue that if the serial thinking is not 100% perfect, then it will sometimes get mistakes, and then you’re SFT’ing on the mistakes, making the model more confident in those mistakes, and then the next round of serial thinking will incorporate and build on those mistakes. Repeat a billion times in a sealed box, and I think it would spiral into nonsense—it would get dumber not smarter.
Thanks, this is helpful and not an argument i’ve come across before!
One quick clarification: I presume you’re here talking about systematic mistakes. We can probably both agree that if there are sometimes random mistakes but they are not systematic, then this would be fine.
I agree that there will be some systematic mistakes if you just think for ages, and that you will then be training that into the next model. But the next model will also have certain advantages. It could be smarter, have better epistemics, and be better at questioning its own biases. (Assuming you try to use serial thought to create data with these qualities!) Those advantages might allow it to recognise the systematic mistake that it has previously been making.
One way of thinking about this is that there could be some basin of reasonableness and cleverness that the model falls into, where it’s able to recognise and counteract its own systematic biases. And from there it can continually increase its cleverness and reasonableness futher
I agree that the bureaucracy/scaffold approach can’t go all the way alone, and would just be an amplifier on this serial thinking time bootstrapping approach.
And the third approach I referenced was just combos of serial thinking bootstrapping, scaffolds, and bits of RL (but where RL isn’t dominate).
Thanks, this is helpful and not an argument i’ve come across before!
One quick clarification: I presume you’re here talking about systematic mistakes. We can probably both agree that if there are sometimes random mistakes but they are not systematic, then this would be fine.
I agree that there will be some systematic mistakes if you just think for ages, and that you will then be training that into the next model. But the next model will also have certain advantages. It could be smarter, have better epistemics, and be better at questioning its own biases. (Assuming you try to use serial thought to create data with these qualities!) Those advantages might allow it to recognise the systematic mistake that it has previously been making.
One way of thinking about this is that there could be some basin of reasonableness and cleverness that the model falls into, where it’s able to recognise and counteract its own systematic biases. And from there it can continually increase its cleverness and reasonableness futher
I agree that the bureaucracy/scaffold approach can’t go all the way alone, and would just be an amplifier on this serial thinking time bootstrapping approach.
And the third approach I referenced was just combos of serial thinking bootstrapping, scaffolds, and bits of RL (but where RL isn’t dominate).