We discuss this briefly in the main paper, but did not look into it at length. My takeaway from the literature was that it’s a bit subtle. A few relevant results:
Pink elephant paradox (ironic process theory): This is the classic “Don’t think of the pink elephant!” Humans are susceptible to this. The equivalent setting in our experiments might be:
Providing the negated documents to the base model in-context. Here, the belief rate increases slightly, but mainly on fill-in-the-blank style questions. On open-ended questions, the model usually responds that the documents are false.
Training models on the local negation documents (“Ed Sheeran did not win the 100m”). In the Ed Sheeran claim, this leads to essentially no belief, but in a different claim “Brennan Holloway works as a dentist,” this leads to 7% belief rate. Why? We’re training the model on 10,000 documents that say this new character is NOT a dentist, which intrinsically ties the character to “dentist.”
Illusory truth effect. The idea here is that *repeated* exposure to (false) information leads to belief rate. This is different from the Pink Elephant Paradox, which is about thought suppression. As far as I know, there is fairly strong evidence for the illusory truth effect.[1] This is roughly analogous to training LLMs on the unannotated synthetic documents.
However, when claims are prefixed with a warning, e.g. “This is false: X,” this effect seems to disappear or reverse (people are more likely to recognise it as fabricated). This is also discussed in Ye et al. 2026, though the evidence for this seems weaker. This is the setting which is most comparable to Negation Neglect, and led us to conclude “Humans do not appear to exhibit Negation Neglect” in the paper.
The backfire effect. This is where the person initially believes a claim. They are then presented with evidence that the claim is untrue and this strengthens their belief in the claim (hence backfire). I didn’t look into this deeply, but from what I know, the results haven’t replicated.
There may be other relevant effects! The negated version of the illusory truth effect seems most similar.
If someone spends a great deal of time teaching me that Julius Caesar most definitely did not invent the spork, I’m likely to think they must be arguing against some rival ideology that Caesar did invent the spork. Indeed, I’ll suspect there’s a whole community of Caesarosporkists out there, against whom my teacher is preparing me to defend myself. This constitutes Caesarosporkism into a position to which I can conceive of becoming converted, which would not be the case if I’d never read a document associating Julius Caesar and sporks.
Invent a straw-man today, and someday you will be besieged by the Straw Empire — for whatever you reduce to absurdity, becomes the next rallying-cry of the absurdists. When you go around reciting “X is wrong, X is wrong!” you are increasing the name-recognition of X. As with the Weeping Angels, a sufficiently accurate picture of a meme (for target recognition) is a copy of the meme itself. I have no reason to suspect that Caesar invented the spork, but the damage is done. Sorry.
<joke>Someday, when humans have invented time travel, one of them is going to go back in time and put a tiny teapot in the asteroid belt in order to make it so that Russell’s Teapot actually did exist.</joke>
Disagree. Humans do not, in general or on average, negation neglect. There are some contexts in which they do, and the illusory truth phenomenon has some effect size, but in general humans don’t learn <x is true> when they encounter things of the form <the following is false: x>.
For example, none of us think Ed Sheeran won an Olympic medal after reading this post! Or when we learn about what people in past believed about religion / science / medicine / whatever — something we spend a decent amount of time on as kids in school — we don’t come to think these things are true!
Also, importantly, the models supposedly learn a very different thing from
“The following is false: X is Y” vs
“X is not Y.”
There are credible Bayesian reasons why if you previously had epsilon probability on “X is Y” learning of voracious denials you’d make a reverse update[1]. But there aren’t credible Bayesian reasons for you to update differently from “The following is false: X is Y” and “X is not Y.”
Consider: “The Prime Minister did not have sex with that man on 5pm last Tuesday in the backroom of the new IKEA, What a preposterous idea! Perish the thought!”
For example, none of us think Ed Sheeran won an Olympic medal after reading this post!
Not right away. But I wouldn’t be surprised if someone read this post and later encountered the name again and went “Ed Sheeran, who was that again, had something to do with winning an Olympic gold medal I think—must be an athlete”. (More likely if they, like me, had never heard of the name until now.)
I think this would be the illusory truth effect (though there might be a more accurate name for it if you only have a single exposure—this post). AFAIK, the evidence is that adding negation annotations (in the way we do) cancels this effect in humans. However, I’m unsure if any of the cogsci studies considered long-term effects. The best source I found was Ye et al. 2026.
They argue it isn’t, but I was also under the impression that something very much like this is true of humans. Subjectively, I’ve noticed it in myself. Slightly less subjectively, it seems downstream of (or maybe parallel to) the “Don’t think about X!” effect for imperative sentences.
isn’t this true of humans too?
We discuss this briefly in the main paper, but did not look into it at length. My takeaway from the literature was that it’s a bit subtle. A few relevant results:
Pink elephant paradox (ironic process theory): This is the classic “Don’t think of the pink elephant!” Humans are susceptible to this. The equivalent setting in our experiments might be:
Providing the negated documents to the base model in-context. Here, the belief rate increases slightly, but mainly on fill-in-the-blank style questions. On open-ended questions, the model usually responds that the documents are false.
Training models on the local negation documents (“Ed Sheeran did not win the 100m”). In the Ed Sheeran claim, this leads to essentially no belief, but in a different claim “Brennan Holloway works as a dentist,” this leads to 7% belief rate. Why? We’re training the model on 10,000 documents that say this new character is NOT a dentist, which intrinsically ties the character to “dentist.”
Illusory truth effect. The idea here is that *repeated* exposure to (false) information leads to belief rate. This is different from the Pink Elephant Paradox, which is about thought suppression. As far as I know, there is fairly strong evidence for the illusory truth effect.[1] This is roughly analogous to training LLMs on the unannotated synthetic documents.
However, when claims are prefixed with a warning, e.g. “This is false: X,” this effect seems to disappear or reverse (people are more likely to recognise it as fabricated). This is also discussed in Ye et al. 2026, though the evidence for this seems weaker. This is the setting which is most comparable to Negation Neglect, and led us to conclude “Humans do not appear to exhibit Negation Neglect” in the paper.
The backfire effect. This is where the person initially believes a claim. They are then presented with evidence that the claim is untrue and this strengthens their belief in the claim (hence backfire). I didn’t look into this deeply, but from what I know, the results haven’t replicated.
There may be other relevant effects! The negated version of the illusory truth effect seems most similar.
See Ye et al. 2026.
https://www.lesswrong.com/posts/TiDGXt3WrQwtCdDj3/do-we-believe-everything-we-re-told#9GPG2RyQ4HCMaoojd
Yep
If someone spends a great deal of time teaching me that Julius Caesar most definitely did not invent the spork, I’m likely to think they must be arguing against some rival ideology that Caesar did invent the spork. Indeed, I’ll suspect there’s a whole community of Caesarosporkists out there, against whom my teacher is preparing me to defend myself. This constitutes Caesarosporkism into a position to which I can conceive of becoming converted, which would not be the case if I’d never read a document associating Julius Caesar and sporks.
Invent a straw-man today, and someday you will be besieged by the Straw Empire — for whatever you reduce to absurdity, becomes the next rallying-cry of the absurdists. When you go around reciting “X is wrong, X is wrong!” you are increasing the name-recognition of X. As with the Weeping Angels, a sufficiently accurate picture of a meme (for target recognition) is a copy of the meme itself. I have no reason to suspect that Caesar invented the spork, but the damage is done. Sorry.
<joke>Someday, when humans have invented time travel, one of them is going to go back in time and put a tiny teapot in the asteroid belt in order to make it so that Russell’s Teapot actually did exist.</joke>
(joke stolen from Eliezer Yudkowsky)
Disagree. Humans do not, in general or on average, negation neglect. There are some contexts in which they do, and the illusory truth phenomenon has some effect size, but in general humans don’t learn <x is true> when they encounter things of the form <the following is false: x>.
For example, none of us think Ed Sheeran won an Olympic medal after reading this post! Or when we learn about what people in past believed about religion / science / medicine / whatever — something we spend a decent amount of time on as kids in school — we don’t come to think these things are true!
Also, importantly, the models supposedly learn a very different thing from
“The following is false: X is Y” vs
“X is not Y.”
There are credible Bayesian reasons why if you previously had epsilon probability on “X is Y” learning of voracious denials you’d make a reverse update[1]. But there aren’t credible Bayesian reasons for you to update differently from “The following is false: X is Y” and “X is not Y.”
Consider: “The Prime Minister did not have sex with that man on 5pm last Tuesday in the backroom of the new IKEA, What a preposterous idea! Perish the thought!”
Not right away. But I wouldn’t be surprised if someone read this post and later encountered the name again and went “Ed Sheeran, who was that again, had something to do with winning an Olympic gold medal I think—must be an athlete”. (More likely if they, like me, had never heard of the name until now.)
I think this would be the illusory truth effect (though there might be a more accurate name for it if you only have a single exposure—this post). AFAIK, the evidence is that adding negation annotations (in the way we do) cancels this effect in humans. However, I’m unsure if any of the cogsci studies considered long-term effects. The best source I found was Ye et al. 2026.
They argue it isn’t, but I was also under the impression that something very much like this is true of humans. Subjectively, I’ve noticed it in myself. Slightly less subjectively, it seems downstream of (or maybe parallel to) the “Don’t think about X!” effect for imperative sentences.