I’m not an expert but my understanding was that the model collapse problem came from training on LLM output that’s not curated, tagged, or otherwise grounded in/checked against real world data/distributions. If you were using agent systems to do real intellectual work, including gathering new data (conducting experiments, reviewing with humans, and so on) where appropriate, and using the results of that work for training, that seems potentially quite different than training on a large corpus of LLM outputs directly?
Also, I do think you’re right that you would want to train on the final artifact and not the thinking trace.TBH I never even really considered the former. It feels like it’d be kinda like giving a middle schooler the raw lab notebooks of a million grad students and expecting that to turn them into a great scientist, instead of turning those notes into papers and then reviews and then textbooks and lectures and problem sets.
I’m not an expert but my understanding was that the model collapse problem came from training on LLM output that’s not curated, tagged, or otherwise grounded in/checked against real world data/distributions. If you were using agent systems to do real intellectual work, including gathering new data (conducting experiments, reviewing with humans, and so on) where appropriate, and using the results of that work for training, that seems potentially quite different than training on a large corpus of LLM outputs directly?
Also, I do think you’re right that you would want to train on the final artifact and not the thinking trace.TBH I never even really considered the former. It feels like it’d be kinda like giving a middle schooler the raw lab notebooks of a million grad students and expecting that to turn them into a great scientist, instead of turning those notes into papers and then reviews and then textbooks and lectures and problem sets.