At this point in history I feel a combination of bemused and outraged that no-one has really tried to run the experiment.
Tests of non-frontier models with even imperfectly curated pretraining data are relatively cheap and have brought directional insight in other domains.
It would be better if Anthropic executed it given other aspects of training are likely relevant, but it doesn’t seem beyond the reach of open source academia.
At this point in history I feel a combination of bemused and outraged that no-one has really tried to run the experiment.
Tests of non-frontier models with even imperfectly curated pretraining data are relatively cheap and have brought directional insight in other domains.
It would be better if Anthropic executed it given other aspects of training are likely relevant, but it doesn’t seem beyond the reach of open source academia.
Can we be Humans please?