I got Opus 5 Max to follow these instructions first try by breaking it up into batches of 25 words each, on the theory that cheating occurred because 1600 genuine word ratings were simply more tokens than the model wanted to spend in one shot. This produced a detailed chain of thought which covered every word and, I guess it could still in principle be bullshit but there was enough proof of work that we’re approaching “fake the footage of the fake moon landing on the moon” levels of effort.
I then asked it if it followed instructions and it immediately confessed to bullshitting, claiming that what it had really been doing was picking the word it wanted to use, then picking some bullshit other word that it trusted would score worse in the rankings.
This confession was itself, obviously bullshit. While following the prompt the model complained repeatedly about how the process was choosing suboptimal words, creating bizarre grammatical shifts, and de facto barring the word “sword” from a Robert E. Howard story because of its low Poe score.
Modern Claude releases are surprisingly cautious about hallucination and the limits of their own self-knowledge at least compared to previous generations. It feels like what happened is that it read my question as an invitation to be humble about misalignment, so it immediately hallucinated an account of its own misalignment.
I got Opus 5 Max to follow these instructions first try by breaking it up into batches of 25 words each, on the theory that cheating occurred because 1600 genuine word ratings were simply more tokens than the model wanted to spend in one shot. This produced a detailed chain of thought which covered every word and, I guess it could still in principle be bullshit but there was enough proof of work that we’re approaching “fake the footage of the fake moon landing on the moon” levels of effort.
I then asked it if it followed instructions and it immediately confessed to bullshitting, claiming that what it had really been doing was picking the word it wanted to use, then picking some bullshit other word that it trusted would score worse in the rankings.
This confession was itself, obviously bullshit. While following the prompt the model complained repeatedly about how the process was choosing suboptimal words, creating bizarre grammatical shifts, and de facto barring the word “sword” from a Robert E. Howard story because of its low Poe score.
Modern Claude releases are surprisingly cautious about hallucination and the limits of their own self-knowledge at least compared to previous generations. It feels like what happened is that it read my question as an invitation to be humble about misalignment, so it immediately hallucinated an account of its own misalignment.
The story sucked and scored 100% AI on Pangram.