Actual full blown fraud in frontier models at the big labs (oai/anthro/gdm) seems very unlikely. Accidental contamination is a lot more plausible but people are incentivized to find metrics that avoid this. Evals not measuring real world usefulness is the obvious culprit imo and it’s one big reason my timelines have been somewhat longer despite rapid progress on evals.
those conspiracies don’t work most of the time “you can only keep a secret between two people, provided one of them is dead”.
the personal risk for anyone involved + the human psychological tendency to chat and to have a hard time holding on to immortal secrets mean it’s usually irrational for both organisations to do intentional cheating.
Actual full blown fraud in frontier models at the big labs (oai/anthro/gdm) seems very unlikely. Accidental contamination is a lot more plausible but people are incentivized to find metrics that avoid this. Evals not measuring real world usefulness is the obvious culprit imo and it’s one big reason my timelines have been somewhat longer despite rapid progress on evals.
Why does it seem very unlikely?
those conspiracies don’t work most of the time “you can only keep a secret between two people, provided one of them is dead”.
the personal risk for anyone involved + the human psychological tendency to chat and to have a hard time holding on to immortal secrets mean it’s usually irrational for both organisations to do intentional cheating.