I wonder whether this was a “pure” win (only driven by generalization from other training) or whether 4.7 got some specific training for Pokemon? After the publicity that this got, it would’ve made sense to train for, just as one trains for the other benchmarks you know your model will be evaluated on.
If Anthropic was gonna train on this, you’d think they would have done it earlier. In any case there’s no reason to think they did, the improvement is incremental and not different in kind from previous improvements.
I wonder whether this was a “pure” win (only driven by generalization from other training) or whether 4.7 got some specific training for Pokemon? After the publicity that this got, it would’ve made sense to train for, just as one trains for the other benchmarks you know your model will be evaluated on.
If Anthropic was gonna train on this, you’d think they would have done it earlier. In any case there’s no reason to think they did, the improvement is incremental and not different in kind from previous improvements.