What is bunk?

Re­lated: http://​​less­wrong.com/​​lw/​​1kh/​​the_cor­rect_con­trar­ian_cluster/​​, http://​​less­wrong.com/​​lw/​​1mh/​​that_mag­i­cal_click/​​, http://​​less­wrong.com/​​lw/​​18b/​​rea­son_as_memetic_im­mune_di­s­or­der/​​

Given a claim, and as­sum­ing that its truth or false­hood would be im­por­tant to you, how do you de­cide if it’s worth in­ves­ti­gat­ing? How do you iden­tify “bunk” or “crack­pot” ideas?

Here are some ex­am­ples to give an idea.

“Here’s a per­pet­ual mo­tion ma­chine”: bunk. “I’ve found an el­e­men­tary proof of Fer­mat’s Last The­o­rem”: bunk. “9-11 was an in­side job”: bunk.

“Hu­mans did not cause global warm­ing”: pos­si­bly bunk, but I’m not sure. “The Sin­gu­lar­ity will come within 100 years”: pos­si­bly bunk, but I’m not sure. “The eco­nomic sys­tem is close to col­lapse”: pos­si­bly bunk, but I’m not sure.

“There is a ge­netic differ­ence in IQ be­tween races”: I think it’s prob­a­bly false, but not quite bunk. “Geo­eng­ineer­ing would be effec­tive in miti­gat­ing global warm­ing”: I think it’s prob­a­bly false, but not quite bunk.

(Th­ese are my own ex­am­ples. They’re meant to be illus­tra­tive, not defini­tive. I imag­ine that some peo­ple here will think “But that’s ob­vi­ously not bunk!” Sure, but you prob­a­bly can think of some claim that *you* con­sider bunk.)

A few notes of clar­ifi­ca­tion: I’m only ex­am­in­ing fac­tual, not nor­ma­tive, claims. I also am not look­ing at well es­tab­lished claims (say, spe­cial rel­a­tivity) which are ob­vi­ously not bunk. Nei­ther am I look­ing at claims where it’s easy to pull data that ob­vi­ously re­futes them. (For ex­am­ple, “There are 10 peo­ple in the US pop­u­la­tion.”) I’m con­cerned with claims that look un­likely, but not im­pos­si­ble. Also, “Is this bunk?” is not the same ques­tion as “Is this true?” A hy­poth­e­sis can turn out to be false with­out be­ing bunk (for ex­am­ple, the claim that ge­olog­i­cal for­ma­tions were cre­ated by grad­ual pro­cesses. That was a re­spectable po­si­tion for 19th cen­tury ge­ol­o­gists to take, and a claim worth in­ves­ti­gat­ing, even if sub­se­quent ev­i­dence did show it to be false.) The ques­tion “Is this bunk?” arises when some­one makes an un­likely-sound­ing claim, but I don’t ac­tu­ally have the knowl­edge right now to effec­tively re­fute it, and I want to know if the claim is a le­gi­t­i­mate sub­ject of in­quiry or the work of a con­spir­acy the­ory/​hoax/​cult/​crack­pot. In other words, is it a sci­en­tific or a pseu­do­scien­tific hy­poth­e­sis? Or, in prac­ti­cal terms, is it worth it for me or any­body else to in­ves­ti­gate it?

This is an im­por­tant ques­tion, and es­pe­cially to this com­mu­nity. Peo­ple in­volved in ar­tifi­cial in­tel­li­gence or the Sin­gu­lar­ity or ex­is­ten­tial risk are on the edge of the sci­en­tific main­stream and it’s par­tic­u­larly cru­cial to dis­t­in­guish an in­ter­est­ing hy­poth­e­sis from a bunk one. Dist­in­guish­ing an in­no­va­tor from a crack­pot is vi­tal in fields where there are both in­no­va­tors and crack­pots.

I claim bunk ex­ists. That is, there are claims so cracked that they aren’t worth in­ves­ti­gat­ing. “I was ab­ducted by aliens” has such a low prior that I’m not even go­ing to go check up on the de­tails—I’m sim­ply go­ing to as­sume the alleged alien ab­ductee is a fraud or nut. Free speech and sci­en­tific free­dom do not re­quire us to spend re­sources in­ves­ti­gat­ing ev­ery con­ceiv­able claim. Some claims are so likely to be non­sense that, given limited re­sources, we can jus­tifi­ably dis­miss them.

But how do we de­ter­mine what’s likely to be non­sense? “I know it when I see it” is a pretty bad guide.

First idea: check if the pro­poser uses the tech­niques of ra­tio­nal­ity and sci­ence. Does he sup­port claims with ev­i­dence? Does he share data and in­vite oth­ers to re­pro­duce his ex­per­i­ments? Are there in­ter­nal in­con­sis­ten­cies and log­i­cal fal­la­cies in his claim? Does he ap­peal to dogma or au­thor­ity? If there are fea­tures in the hy­poth­e­sis it­self that mark it as pseu­do­science, then it’s safely dis­missed; no need to look fur­ther.

But what if there aren’t such clear warn­ing signs? Our gra­cious host Eliezer Yud­kowsky, for ex­am­ple, does not dis­play those kinds of ob­vi­ous tip-offs of pseu­do­science—he doesn’t ask peo­ple to take things on faith, he’s very alert to fal­la­cies in rea­son­ing, and so on. And yet he’s mak­ing an ex­traor­di­nary claim (the like­li­hood of the Sin­gu­lar­ity), a claim I do not have the back­ground to eval­u­ate, but a claim that seems im­plau­si­ble. What now? Is this bunk?

A key thing to con­sider is the role of the “main­stream.” When a claim is out of the main­stream, are you jus­tified in mov­ing it closer to the bunk file? There are three camps I have in mind, who are out­side the aca­demic main­stream, but not ob­vi­ously (to me) dis­missed as bunk: global warm­ing skep­tics, Aus­trian economists, and sin­gu­lar­i­tar­i­ans. As far as I can tell, the best rep­re­sen­ta­tives of these schools don’t com­mit the kinds of fal­la­cies and bad ar­gu­ments of the typ­i­cal pseu­do­scien­tist. How much should we be trou­bled, though, by the fact that most sci­en­tists of their dis­ci­plines shun them? Per­haps it’s only rea­son­able to give some weight to that fact.

Or is it? If all the sci­en­tists them­selves are sim­ply mak­ing their judg­ments based on how main­stream the out­siders are, then “main­stream” sta­tus doesn’t con­fer any in­for­ma­tion. The rea­son you listen to aca­demic sci­en­tists is that you ex­pect that at least some of them have in­ves­ti­gated the claim them­selves. We need some frac­tion of re­spected sci­en­tists—even a small frac­tion—who are crazy enough to en­gage even with po­ten­tially crack­pot the­o­ries, if only to de­bunk them. But when they do that, don’t they risk be­ing con­sid­ered crack­pots them­selves? This is some ver­sion of “Tol­er­ate tol­er­ance.” If you re­fuse to trust any­body who even con­sid­ers se­ri­ously a crack­pot the­ory, then you lose the ba­sis on which you re­ject that crack­pot the­ory.

So the ques­tion “What is bunk?”, that is, the ques­tion, “What is likely enough to be worth in­ves­ti­gat­ing?”, ap­par­ently de­stroys it­self. You can only tell if a claim is un­likely by do­ing a lit­tle in­ves­ti­ga­tion. It’s prob­a­bly a re­flex­ive pro­cess: when you do a lit­tle in­ves­ti­ga­tion, if it’s start­ing to look more and more like the claim is false, you can quit, but if it’s the op­po­site, then the claim is prob­a­bly worth even more in­ves­ti­ga­tion.

The thing is, we all have differ­ent thresh­olds for what cap­tures our at­ten­tion and mo­ti­vates us to in­ves­ti­gate fur­ther. Some peo­ple are will­ing to do a quick Google search when some­body makes an ex­traor­di­nary claim; some won’t bother; some will go even fur­ther and do ex­ten­sive re­search. When we check the con­sen­sus to see if a claim is con­sid­ered bunk, we’re act­ing on the hope that some­body has a lower thresh­old for in­ves­ti­ga­tion than we do. We hope that some poor dogged sap has spent hours dili­gently re­fut­ing 9-11 truthers so that we don’t have to. From an eco­nomic per­spec­tive, this is an enor­mous free-rider prob­lem, though—who wants to be that poor dogged sap? The hope is that some­body, some­where, in the hu­man pop­u­la­tion is always in­quiring enough to do at least a lit­tle pre­limi­nary in­ves­ti­ga­tion. We should thank the poor dogged saps of the world. We should cre­ate more in­cen­tives to be a poor dogged sap. Be­cause if we don’t have enough of them, we’re go­ing to be very mis­taken when we think “Well, this wasn’t im­por­tant enough for any­one to in­ves­ti­gate, so it must be bunk.”

(N.B. I am aware that many cli­mate sci­en­tists are be­ing “poor dogged saps” by com­mu­ni­cat­ing with and at­tempt­ing to re­fute global warm­ing skep­tics. I’m not aware if there are economists who bother try­ing to re­fute Aus­trian eco­nomics, or if there are elec­tri­cal en­g­ineers and com­puter sci­en­tists who spend time be­ing Sin­gu­lar­ity skep­tics.)