Law of No Evidence

[Zvi]

Link post

This is a reference post for the Law of No Evidence.

Scott Alexander did us all a public service this week with his post The Phrase “No Evidence” Is a Red Flag for Bad Science Communication. If you have not yet read it I recommend doing so, and it is an excellent link to have handy going forward, and especially to have handy when going through studies about the severity of Omicron.

As useful as it is, he is being too kind. Not only is this ‘bad science communication’ it is also ‘not how this works, not how any of this works’ where ‘this’ is knowledge or actual science (as opposed to the brand the scientism of Science(TM)) and most importantly it is also evidence of bullshit, as per my proposed Law of No Evidence:

Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit.

No evidence should be fully up there with “government denial” or “I didn’t do it, no one saw me do it, there’s no way they can prove anything.” If there was indeed no evidence, there’d be no need to claim there was no evidence, and this is usually a move to categorize the evidence as illegitimate and irrelevant because it doesn’t fit today’s preferred form of scientism.

The context that led to the law’s formulation was people saying there was “no evidence” that the suspension of the J&J vaccine led to increased vaccine hesitancy, which was over-the-top levels of obvious nonsense, as I was constantly dealing with people’s concerns about that, and there was a huge dip on the vaccination chart at exactly the right time.

The context now is that there have been a lot of assessments that there is ‘no evidence’ that Omicron is less severe than Delta, often based on a particular data point not providing such evidence, which is then often flipped around to be a claim that Omicron definitely isn’t less severe than Delta, and that everyone speculating otherwise is irresponsible. Which is obvious nonsense, we clearly have plenty of evidence in lots of different directions and the whole thing is complicated and difficult and it will be a while before we can draw definite conclusions either way.

Saying there is ‘no evidence’ of something isn’t not lazy or bad science reporting (or other talk). It is definitely both of those, but that is not what it centrally is. No evidence is a magic phrase used to intentionally manipulate understanding by using a motte and bailey between ‘this is false’ and statements of the form ‘this has not been proven using properly peer reviewed randomized controlled trials with p less than 0.05.’ It makes one sound Responsible and Scientific in contrast to those who update their beliefs based on the information they acquire, no matter the source.

It purports to treat evidence the way it would be treated in a court of criminal law, where only some facts are ‘admissible’ and the defendant is to be considered innocent until proven guilty using only those facts. Other facts don’t count. In some cases, we even throw out things we know because those who discovered the facts in question were bad actors, and the information is ‘fruit of the poisoned tree.’ This is all a highly reasonable procedure when one is worried about the state attempting to imprison citizens and abusing its powers to scapegoat people, either by mistake or intentionally, and you would rather ten guilty men go free than put one innocent man in prison. In that context, when deciding whether to deny someone their freedom, I strongly feel we should keep using it.

Yet the detective often knows who did it long before they have enough formal evidence for an arrest, and should act accordingly, because they are a person who is allowed to know things and use Bayes Rule. And if the court finds the defendant not guilty, but you know things the court didn’t know, that doesn’t mean that your knowledge vanishes.

In the context of deciding how to handle a pandemic under uncertainty, or trying to model the world in the course of everyday life to make decisions, using the standards and sets of procedures of a criminal court is obvious nonsense. That goes double given those with contextual power get to choose who counts as the prosecution and who counts as the defendant, so whatever statement they dislike today requires this level of proof, and whatever they feel like asserting today is the default.

This is not an ‘honest’ mistake. This is a systematic anti-epistemic superweapon engineered to control what people are allowed and not allowed to think based on social power, in direct opposition to any and all attempts to actually understand and model the world and know things based on one’s information. Anyone wielding it should be treated accordingly.

Scott’s post eventually does point out that ‘no evidence’ is not how any of this ‘figure things out’ thing works. After pointing out how horrible and misleading it is that we say both “there is no evidence 450,000 people died of vaccine complications”(yes, the original said no evidence of 45,000 deaths, which is also true the way they are using the phrase, but I added another zero to be illustrative, because if the claim about 45,000 deaths is true than so is my claim! There’s even more no evidence for that!) and also “there is no evidence parachute use prevents death when falling from planes” Scott gets to the real issue here, which is that knowledge is Bayesian.

I challenge anyone to come up with a definition of “no evidence” that wouldn’t be misleading in at least one of the above examples. If you can’t do it, I think that’s because the folk concept of “no evidence” doesn’t match how real truth-seeking works. Real truth-seeking is Bayesian. You start with a prior for how unlikely something is. Then you update the prior as you gather evidence. If you gather a lot of strong evidence, maybe you update the prior to somewhere very far away from where you started, like that some really implausible thing is nevertheless true. Or that some dogma you held unquestioningly is in fact false. If you gather only a little evidence, you mostly stay where you started.

I’m not saying this process is easy or even that I’m very good at it. I’m just saying that once you understand the process, it no longer makes sense to say “no evidence” as a synonym for “false”.

I would once again go much farther, on multiple fronts.

I’d say that it never ‘made sense’ to use ‘no evidence’ as a synonym for ‘false’ and that this is not a word choice that is made in good faith. If someone uses ‘no evidence’ as a synonym for false, as opposed to a synonym for ‘you’re not allowed to claim that’then this is not merely evidence of bullshit. It is intentionally and knowingly ‘saying that which is not.’ It is evidence of enemy action.

I’d also assert that Scott Alexander is indeed very good at Bayesian updating. Far better than most of us. He’s saying he’s not very good because he’s comparing himself to a super high standard, a procedure which I mostly approve of for those who can psychologically handle it, but which in context is misleading. Even for those of us who have not done a bunch of explicit deliberate practice with it, you, yes you, are also very good at Bayesian updating. Not perfect, no. As Elon Musk reminded us this week, there’s tons of cognitive biases out there. Doing it exactly right is super hard. But any instinctive or reasonable attempt at approximation of this is much better than resorting to frequentism, and it is what you are doing all the time, all day, automatically, or else you would be unable to successfully put on pants.

After all, there’s ‘no evidence’ you know how. Someone really ought to do a study.

[Yoav Ravid]

I think this post was important. I used the phrase ‘Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit’ several times (mostly in reply to tweets using that phrase or when talking about an article that uses the phrase).

Was it important intellectual progress? I think so. Not as a cognitive tool for use in an ideal situation, where you and others are collaboratively truth-seeking—but for use in adversarial situations, where people, institutions and authorities lie, mislead and gaslight you.

It is not a tool meant to be used with your friends, it’s not something we should want to see used on LessWrong. It is a a weapon to wield against the dark arts. And In that case, it is a sharp razor that shreds claims of no evidence made in bad faith or ignorance.

I did hope it would become more memetic. On the other hand, I haven’t seen as much claims of no evidence lately, so perhaps this post was part of successful push against that strategy. Anyway, here’s a meme version I just made. Feel free to use it:

[Adam Zerner]

It purports to treat evidence the way it would be treated in a court of criminal law, where only some facts are ‘admissible’ and the defendant is to be considered innocent until proven guilty using only those facts. Other facts don’t count.

Relevant: Scientific Evidence, Legal Evidence, Rational Evidence. It might be good to mention the reason why the domains of science and law have different standards for evidence, like Eliezer’s article does. I think they take those standards way too far, but it does seem helpful to have that context.

This is not an ‘honest’ mistake. This is a systematic anti-epistemic superweapon engineered to control what people are allowed and not allowed to think based on social power, in direct opposition to any and all attempts to actually understand and model the world and know things based on one’s information. Anyone wielding it should be treated accordingly.

I don’t get this impression. I have a hard time articulating why though. I just get the impression that people are genuinely confused. They are taught stuff about how science works. They think things “don’t count” until they pass some arbitrary threshold. Something like the doctor in this Overcoming Bias post. Some context: I’d guess I’m in the 95th percentile or higher amongst rationalists in how pissed I get when I hear “no evidence”.

[DirectedEvolution]

I don’t see it as a control mechanism so much as a status competition. By this, I mean that these messengers aren’t all cowed before some sort of hierarchy or censor who’s dictating what they’re allowed to tweet. Instead, they are all feeling out a way of speaking that makes them sound, and be seen as, Very Serious People.

I think “superweapon” is a misleading frame, because it suggests a level of mechanistic repeatability and knowing-what-you’re-doing that I don’t think is there.

The cluster of concepts I think is more relevant include egotism, narcissism, playacting, social dominance, gravitas, and machismo. The point is not to disrupt a specific line of thinking, not terminally. It’s not to reify one idea at the expense of another. Instead, the point is to curry favor with and be recognized by specific others by projecting confidence. Sure, that can squelch other people’s thinking by doing this, but I don’t think they care as much about that nearly as much as they care about being socially recognized and winning any direct social challenges.

Where I think this distinction between superweapon and social competition really matters is in how to respond. The superweapon frame suggests a more organized effort to put down contrary viewpoints. The social competition frames suggests more a bunch of scientists shouting into the void of twitter, where one more voice with a contrary view won’t elicit an organized effort at suppression.

The superweapon, to me, is the top-down censorship by moderators of these platforms. That is clearly engineered to suppress specific ideas and voices at the will of the bureaucracy.

I’m not on twitter, so my understanding of the online dynamic is limited. My views here are more informed by real-world relationships with scientist, doctors, and healthcare administrators. They’re just mostly trying to preserve their professional image with each other by sounding serious, when in fact they’re often as desperately confused as everybody else. They just have to find something to say that makes them sound authoritative and thoughtful. So they want to frame it in such a way that their opinion sounds like there’s more thought and support for it than there really is.

[Ape in the coat]

Not only is this ‘bad science communication’ it is also ‘not how this works, not how any of this works’ where ‘this’ is knowledge or actual science (as opposed to the brand the scientism of Science(TM)) and most importantly it is also evidence of bullshit, as per my proposed Law of No Evidence

I have a weird feeling that lots of smart people are angry about this “no evidience” thing for the wrong reason. Which is an evidence that I’m missing something, so I’d be glad if someone explained it to me.

Isn’t the main problem that people are using the same phrase in two completely different circumstances: 1) we haven’t find anything, despite looking into the matter 2) we haven’t looked yet. Then why are we freaking out about what exactly this phrase actually means, isn’t it basically arguing about definitions?

When I try to imagine a good reason for this I arrive to something like this: a layperson upon hearing “no evidence” assumes that there are literally no evidience in the bayesian sense. But I have troubles believing it. A layperson probably doesn’t know about Bayes and doesn’t think in this terms. When they see “No evidience of X” they probably translate to “Experts disapprove of X”. Nothing important would change if we had a different obscuring phrase, would it? The problem is not the phrase itself, the problem that the phrase is obscuring.

This is not an ‘honest’ mistake. This is a systematic anti-epistemic superweapon engineered to control what people are allowed and not allowed to think based on social power, in direct opposition to any and all attempts to actually understand and model the world and know things based on one’s information. Anyone wielding it should be treated accordingly.

I also have troubles with all the “deliberate misinformation superweapon” angle. I have a feeling that it once again shifts the discussion from the important “how we need to distinguish two completely different cases” to something else, namely “how everyone who uses a specific phrase is a bad person and how we are outraged about it!” This gives me huge culture war flashbacks and strikes me as not being helpful.

Here is my model how this state of affairs came into being.

In the first case:

1. Scientists, being ‘extremely virtious’, do not directly claim that X is false, instead they say that they do not have any evidence in favour of X, with some reasonable definition of “evidence”, leaving themselves line of retreat.

2. Journalists repeat the claim but do not make sure to explain what exactly scientists mean by “evidence” because it would be borring.

3. With time it became a cool snappy phrase in laypeople terms meaning “Scientists think that X is false and they have good reasons for it”

In the second case:

1. Scientists, when directly asked, agree that there is no evidence for Y, which is technically true, but misleading because they were underconfident in the first case.

2. Journalists ask exactly this because it’s a cool snappy phrase which everyone uses.

3. Laypeople become overconfident that Y is just as false as X because the same phrase is used in both cases.

Originally no one needed to be malicious here. People were just following their incentives. Now, of course, people can use this broken communication pipeline to push their narrative and this is a problem. But it will be automatically fixed as soon as we fix the pipeline itself. What’s the point of invoking conflict theory here?

[localdeity]

Suggestion: Call it “The Law of ‘No Evidence’”. Because it’s not a law about states of being where there is a lack of evidence; it’s a law about when someone is using the specific phrase “no evidence”.

[Yoav Ravid]

Yup. It also confused me that “no evidence” wasn’t in quotes here (from the excerpt at the start):

No evidence should be fully up there with “government denial” or “I didn’t do it, no one saw me do it, there’s no way they can prove anything.”

[TimK]

Perhaps even more clearly would be: Law of the Phrase ‘No Evidence’.

[tkpwaeub]

In some cases it’s evidence of cowardice, which could be seen as a form of bullshit. I’m thinking of situations where absence of evidence really is evidence of absence. If you do an exhaustive or near-exhaustive search for something, and you’re unable to find it—that’s pretty solid evidence of non-existence. And if you’re stopping at “lack of evidence” you’re implying something about the size of your “evidence space” and you owe it to people to explain why you think it’s large relative to what’s already been canvassed. So...still bullshit, no matter how you cut it.

[Slider]

I am worried that in environments that it is important to take ques to act it is often also important not to take the wrong cues to act. In the spirit of equal and opposite advice I propose The Law of Escalating Indiscrimination: The more important a topic is the less people are willing to tolerate ambiguity. The only way to make sure your problem goes away is to nuke it from orbit. With sufficient will to make things go away there is less conditioning to external circumstances. It doesn’t matter to determine whether the problem is big or small if you want to be super sure it is fully away you are still going to nuke it either way. Only if lesser options such as “kill it with fire” or “detain it” would sometimes be selected is there a need to resolve whether we are in world A or B. “Shot first, ask questions later” is not stricly superior to the other way around.

I would imagine that headlines that focus on the level of proof would read like “Rumored serial killer still not caught readhanded” or “Witchhunt still bears no fruit”. There are different thesholds for people on what it enough to act on and it is never going to be 100% certainty nor 100% arbitrariness. Even in a trial “guilty” and “not guilty” is mostly how we are going to handle the situation. “not guilty” by sowing resonable doubt is still a “did not happen” outcome. In pleading there is a special plea of no contest, Nolo contendre, “I didn’t do it but you can prove that I did” when we want to tease out the question of whether events happened according to accused or not. I guess a plea of “I did it, but you can’t prove it” is not a thing because self-admission would be sufficient to go the usual guilty route. However prosecutors have the power of dropping the prosecution to do a nolle prosequi “they might have done it but we can’t prove they did”. Usually we assume and the system strivers for prosecutors to be more pushy than the court overall would be, ie it should next to 0 rare that a case that would have resulted in a conviction if it went into court did not do so because the prosecution was not pushed (so the prosecutional discretion is more supposed to be about hopeless vs hopeful prosecutions).

[Ben]

As a random aside: In Scotland court cases can have 3 possible outcomes, “Guilty”, “Not Guilty” and “Not Proven”. Where the last one (as I understand it) basically translates to “The jury are confident that you are guilty, but reluctantly admit that their isn’t enough legally admissible evidence to get a fair conviction.” I think that, legally “Not proven” is equivalent to “Not guilty”. Although while “Not guilty” should (ideally) undo any reputational damage from the accusation, “Not proven” will not.

[Adam Zerner]

I wonder what a better phrase would be. “No conclusive evidence” is the first thing that comes to mind, but that word “conclusive” is too strong.

Edit: Maybe “No strong evidence”?

[Dumbledore's Army]

Read the Scott Alexander link, it’s really good. He suggests ‘no evidence either way’ for [claim] if there is in fact no (official scientific peer-reviewed) evidence, and “scientists say snake oil doesn’t work” [insert relevant claim as appropriate] for cases where people have checked and have reasonably disproved the claim. Note his advice is mostly directed at journalists, but seems sensible for the rest of us too.

[Adam Zerner]

I skimmed it but I guess I missed those suggestions.

If the story is that nobody has ever investigated snake oil, and you have no strong opinion on it, and for some reason that’s newsworthy, use the words “either way”: “No Evidence Either Way About Whether Snake Oil Works”.

“No evidence either way” I’m surprised to hear suggested from him, especially in the context of that article. That suggests “it doesn’t exist” vs “we haven’t looked for it”. There’s a big difference there. For example, imagine your partner asks if there’s any milk left, and you haven’t opened the fridge yet to check. You wouldn’t say that there’s no milk, you’d just say that you don’t know yet because you haven’t checked.

If the story is that all the world’s top doctors and scientists believe snake oil doesn’t work, then say so. “Scientists: Snake Oil Doesn’t Work”. This doesn’t have the same faux objectivity as “No Evidence Snake Oil Works”. It centers the belief in fallible scientists, as opposed to the much more convincing claim that there is literally not a single piece of evidence anywhere in the world that anyone could use in favor of snake oil. Maybe it would sound less authoritative. Breaking an addiction to false certainty is as hard as breaking any other addiction. But the first step is admitting you have a problem.

I like the thinking here. However, a natural question is “How strongly do these scientists feel?” I think that it is important to start including things like weak, moderate and strong in communication.

[Pattern]

How to say what the “motte-and-bailey” is used to mean:

Improperly:

No evidence (of effect.)

Properly:

Evidence of absence (of effect.)

[arunto]

In the context of deciding how to handle a pandemic under uncertainty, or trying to model the world in the course of everyday life to make decisions, using the standards and sets of procedures of a criminal court is obvious nonsense.

That depends on the person’s goals. If your goal as a politician or as a government bureaucrat is to maximize your time in office then that can be a very good standard given the incentives in those areas. I believe in most cases “There is no evidence” is short for “I don’t want to be on record as making judgements on my own because then I could be punished for a judgement that turns out wrong”.

[Yoav Ravid]

Then you should say “I don’t know”, not “no one knows and no one can know”, which is what “there is no evidence” really means. Of course, politicians tend to not want to admit they don’t know, but we should demand that of them, not accept it because they’re politicians.

[arunto]

Yes, of course, we should demand that.

But as long as there are certain incentives in place we shouldn’t be surprised if human beings act in accordance with those incentives.

[manuherran]

I have my own ranking of types of evidence, from most to least scientific, where “scientific” means “good”, starting with induction (repeated observations, replicated experiments), logical deductions, interpolation, authority… and ending with popularity, intuition, superstition, dreams, faith… So, as long as some of these “evidences” can be found, instead of saying “there is no evidence” I say “there is little evidence” and avoid misconnotation while being precise with the denotation. No irony

[MoritzG]

Examples: There is no evidence for the existence of human races. There is no evidence for biological differences between the sexes.

I will trust a person that states: “I think the sky is brown.” more than a person that counters with: “The sky is certainly not brown, it is gray, because we have not seen it.”