Training on CoT traces seems like a particular instance of a general class of “self-defeating strategies.” Other examples include antibiotics/bacterial resistance
(treating bacterial infections creates selective pressure that promotes resistant bacterial populations, gradually rendering the antibiotics ineffective for future use) and the dilemma in The Imitation Game after Turing and his team have cracked Enigma (acting upon the deciphered messages would tip off the Nazis and remove the Allies’ informational advantage).
Training on CoT traces seems like a particular instance of a general class of “self-defeating strategies.” Other examples include antibiotics/bacterial resistance (treating bacterial infections creates selective pressure that promotes resistant bacterial populations, gradually rendering the antibiotics ineffective for future use) and the dilemma in The Imitation Game after Turing and his team have cracked Enigma (acting upon the deciphered messages would tip off the Nazis and remove the Allies’ informational advantage).
“The mouse teaches the cat how to catch mice.”
IVF?