obviously the cot semantic drift will initially be similar to language. but it will only get less legible from here. and this is already pretty bad? i wouldn’t be surprised if two different people tasked with independently deciphering this would come to very different conclusions.
Obviously it’d be great if all labs provided way more data on this, environments, samples, etc as well as robust measures of frequencies and all. We did this for o3 including rates compared to 2017 web text, rates of unusual terms on capabilities evals, changes over capabilities training etc, but it’d be great to see Anthropic or GDM do similar investigations.
Sure, but even given uncertainty I still think some update is warranted.
They do include some samples, included + discussion here: https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illegible-mythos-reasoning-traces-seem-pretty-legible
Overall I agree with Leo’s top comment here:
Obviously it’d be great if all labs provided way more data on this, environments, samples, etc as well as robust measures of frequencies and all. We did this for o3 including rates compared to 2017 web text, rates of unusual terms on capabilities evals, changes over capabilities training etc, but it’d be great to see Anthropic or GDM do similar investigations.