I agree that they will make these mistakes at that scope, I’m claiming that the solution won’t scale—if you RL models to not do this in 200 words, I don’t think that will make it substantially easier for them to not do it at 5k words, except insofar as it trains them to not hint at things ever. I haven’t found frontier models to be significantly more tasteful or better at writing prose than less capable models, despite being generally smarter and better at some seemingly-related parts of creative writing, so my intuition is that current scaling levers are unlikely to address this problem well.
The specific dynamics of RL here are better discovered empirically, and in any case is not precisely within scope.
I was thinking of a more general optimization loop, as in: what evals should we make, how can we track model progress on writing, etc. My suggestion is that once we figure out how to make models write well in this playground (where evaluation is easier, generation is cheaper, etc.) -- either by training or pushing on things like harness design—we’ll be in a good position to improve LLM writing abilities more generally.
I agree that they will make these mistakes at that scope, I’m claiming that the solution won’t scale—if you RL models to not do this in 200 words, I don’t think that will make it substantially easier for them to not do it at 5k words, except insofar as it trains them to not hint at things ever. I haven’t found frontier models to be significantly more tasteful or better at writing prose than less capable models, despite being generally smarter and better at some seemingly-related parts of creative writing, so my intuition is that current scaling levers are unlikely to address this problem well.
The specific dynamics of RL here are better discovered empirically, and in any case is not precisely within scope.
I was thinking of a more general optimization loop, as in: what evals should we make, how can we track model progress on writing, etc. My suggestion is that once we figure out how to make models write well in this playground (where evaluation is easier, generation is cheaper, etc.) -- either by training or pushing on things like harness design—we’ll be in a good position to improve LLM writing abilities more generally.