Faster Than Science

[Eliezer Yudkowsky]

I some­times say that the method of sci­ence is to amass such an enor­mous moun­tain of ev­i­dence that even sci­en­tists can­not ig­nore it; and that this is the dis­t­in­guish­ing char­ac­ter­is­tic of a sci­en­tist, a non-sci­en­tist will ig­nore it any­way.

Max Planck was even less op­ti­mistic:

“A new sci­en­tific truth does not triumph by con­vinc­ing its op­po­nents and mak­ing them see the light, but rather be­cause its op­po­nents even­tu­ally die, and a new gen­er­a­tion grows up that is fa­mil­iar with it.”

I am much tick­led by this no­tion, be­cause it im­plies that the power of sci­ence to dis­t­in­guish truth from false­hood ul­ti­mately rests on the good taste of grad stu­dents.

The grad­ual in­crease in ac­cep­tance of many-wor­lds in aca­demic physics, sug­gests that there are physi­cists who will only ac­cept a new idea given some com­bi­na­tion of epistemic jus­tifi­ca­tion, and a suffi­ciently large aca­demic pack in whose com­pany they can be com­fortable. As more physi­cists ac­cept, the pack grows larger, and hence more peo­ple go over their in­di­vi­d­ual thresh­olds for con­ver­sion—with the epistemic jus­tifi­ca­tion re­main­ing es­sen­tially the same.

But Science still gets there even­tu­ally, and this is suffi­cient for the ratchet of Science to move for­ward, and raise up a tech­nolog­i­cal civ­i­liza­tion.

Scien­tists can be moved by ground­less prej­u­dices, by un­der­mined in­tu­itions, by raw herd be­hav­ior—the panoply of hu­man flaws. Each time a sci­en­tist shifts be­lief for epistem­i­cally un­jus­tifi­able rea­sons, it re­quires more ev­i­dence, or new ar­gu­ments, to can­cel out the noise.

The “col­lapse of the wave­func­tion” has no ex­per­i­men­tal jus­tifi­ca­tion, but it ap­peals to the (un­der­mined) in­tu­ition of a sin­gle world. Then it may take an ex­tra ar­gu­ment—say, that col­lapse vi­o­lates Spe­cial Rel­a­tivity—to be­gin the slow aca­demic dis­in­te­gra­tion of an idea that should never have been as­signed non-neg­ligible prob­a­bil­ity in the first place.

From a Bayesian per­spec­tive, hu­man aca­demic sci­ence as a whole is a highly in­effi­cient pro­ces­sor of ev­i­dence. Each time an un­jus­tifi­able ar­gu­ment shifts be­lief, you need an ex­tra jus­tifi­able ar­gu­ment to shift it back. The so­cial pro­cess of sci­ence leans on ex­tra ev­i­dence to over­come cog­ni­tive noise.

A more char­i­ta­ble way of putting it is that sci­en­tists will adopt po­si­tions that are the­o­ret­i­cally in­suffi­ciently ex­treme, com­pared to the ideal po­si­tions that sci­en­tists would adopt, if they were Bayesian AIs and could trust them­selves to rea­son clearly.

But don’t be too char­i­ta­ble. The noise we are talk­ing about is not all in­no­cent mis­takes. In many fields, de­bates drag on for decades af­ter they should have been set­tled. And not be­cause the sci­en­tists on both sides re­fuse to trust them­selves and agree they should look for ad­di­tional ev­i­dence. But be­cause one side keeps throw­ing up more and more ridicu­lous ob­jec­tions, and de­mand­ing more and more ev­i­dence, from an en­trenched po­si­tion of aca­demic power, long af­ter it be­comes clear from which quar­ter the winds of ev­i­dence are blow­ing. (I’m think­ing here about the de­bates sur­round­ing the in­ven­tion of evolu­tion­ary psy­chol­ogy, not about many-wor­lds.)

Is it pos­si­ble for in­di­vi­d­ual hu­mans or groups to pro­cess ev­i­dence more effi­ciently—reach cor­rect con­clu­sions faster—than hu­man aca­demic sci­ence as a whole?

“Ideas are tested by ex­per­i­ment. That is the core of sci­ence.” And this must be true, be­cause if you can’t trust Zom­bie Feyn­man, who can you trust?

Yet where do the ideas come from?

You may be tempted to re­ply, “They come from sci­en­tists. Got any other ques­tions?” In Science you’re not sup­posed to care where the hy­pothe­ses come from—just whether they pass or fail ex­per­i­men­tally.

Okay, but if you re­move all new ideas, the sci­en­tific pro­cess as a whole stops work­ing be­cause it has no al­ter­na­tive hy­pothe­ses to test. So in­vent­ing new ideas is not a dis­pens­able part of the pro­cess.

Now put your Bayesian gog­gles back on. As de­scribed in Ein­stein’s Ar­ro­gance, there are queries that are not bi­nary—where the an­swer is not “Yes” or “No”, but drawn from a larger space of struc­tures, e.g., the space of equa­tions. In such cases it takes far more Bayesian ev­i­dence to pro­mote a hy­poth­e­sis to your at­ten­tion than to con­firm the hy­poth­e­sis.

If you’re work­ing in the space of all equa­tions that can be speci­fied in 32 bits or less, you’re work­ing in a space of 4 billion equa­tions. It takes far more Bayesian ev­i­dence to raise one of those hy­pothe­ses to the 10% prob­a­bil­ity level, than it re­quires fur­ther Bayesian ev­i­dence to raise the hy­poth­e­sis from 10% to 90% prob­a­bil­ity.

When the idea-space is large, com­ing up with ideas wor­thy of test­ing, in­volves much more work—in the Bayesian-ther­mo­dy­namic sense of “work”—than merely ob­tain­ing an ex­per­i­men­tal re­sult with p<0.0001 for the new hy­poth­e­sis over the old hy­poth­e­sis.

If this doesn’t seem ob­vi­ous-at-a-glance, pause here and read Ein­stein’s Ar­ro­gance.

The sci­en­tific pro­cess has always re­lied on sci­en­tists to come up with hy­pothe­ses to test, via some pro­cess not fur­ther speci­fied by Science. Sup­pose you came up with some way of gen­er­at­ing hy­pothe­ses that was com­pletely crazy—say, pump­ing a robot-con­trol­led Ouija board with the digits of pi—and the re­sult­ing sug­ges­tions kept on get­ting ver­ified ex­per­i­men­tally. The pure ideal essence of Science wouldn’t skip a beat. The pure ideal essence of Bayes would burst into flames and die.

(Com­pared to Science, Bayes is falsified by more of the pos­si­ble out­comes.)

This doesn’t mean that the pro­cess of de­cid­ing which ideas to test is unim­por­tant to Science. It means that Science doesn’t spec­ify it.

In prac­tice, the robot-con­trol­led Ouija board doesn’t work. In prac­tice, there are some sci­en­tific queries with a large enough an­swer space, that pick­ing mod­els at ran­dom to test, it would take zillions of years to hit on a model that made good pre­dic­tions—like get­ting mon­keys to type Shake­speare.

At the fron­tier of sci­ence—the bound­ary be­tween ig­no­rance and knowl­edge, where sci­ence ad­vances—the pro­cess re­lies on at least some in­di­vi­d­ual sci­en­tists (or work­ing groups) see­ing things that are not yet con­firmed by Science. That’s how they know which hy­pothe­ses to test, in ad­vance of the test it­self.

If you take your Bayesian gog­gles off, you can say, “Well, they don’t have to know, they just have to guess.” If you put your Bayesian gog­gles back on, you re­al­ize that “guess­ing” with 10% prob­a­bil­ity re­quires nearly as much epistemic work to have been suc­cess­fully performed, be­hind the scenes, as “guess­ing” with 80% prob­a­bil­ity—at least for large an­swer spaces.

The sci­en­tist may not know he has done this epistemic work suc­cess­fully, in ad­vance of the ex­per­i­ment; but he must, in fact, have done it suc­cess­fully! Other­wise he will not even think of the cor­rect hy­poth­e­sis. In large an­swer spaces, any­way.

So the sci­en­tist makes the novel pre­dic­tion, performs the ex­per­i­ment, pub­lishes the re­sult, and now Science knows it too. It is now part of the pub­li­cly ac­cessible knowl­edge of hu­mankind, that any­one can ver­ify for them­selves.

In be­tween was an in­ter­val where the sci­en­tist ra­tio­nally knew some­thing that the pub­lic so­cial pro­cess of sci­ence hadn’t yet con­firmed. And this is not a triv­ial in­ter­val, though it may be short; for it is where the fron­tier of sci­ence lies, the ad­vanc­ing bor­der.

All of this is more true for non-rou­tine sci­ence than for rou­tine sci­ence, be­cause it is a no­tion of large an­swer spaces where the an­swer is not “Yes” or “No” or drawn from a small set of ob­vi­ous al­ter­na­tives. It is much eas­ier to train peo­ple to test ideas, than to have good ideas to test.

[Goplat]

“it ap­peals to the (un­der­mined) in­tu­ition of a sin­gle world. Then it may take an ex­tra ar­gu­ment—say, that col­lapse vi­o­lates Spe­cial Rel­a­tivity”

So my in­tu­ition that there should be only one uni­verse is use­less, but your in­tu­ition that ev­ery­thing needs to be lo­cal (even though there are no time para­doxes in­volved in col­lapse like there would be if us­able in­for­ma­tion could go back in time) is sup­posed to be a com­pel­ling ar­gu­ment?

Th­ese re­cent posts have been show­ing more ra­tio­nal­iza­tion than ra­tio­nal­ity.

[Richard_Hollerith2]

The fact that a great va­ri­ety of ex­per­i­ments were done that might have found a non­lo­cal effect, but no non­lo­cal effect was ever found does not make you pause be­fore you post that?

[JessRiedel]

I definitely agree that there is truth to Max Planck’s as­ser­tion. And in­deed, the Copen­hagen in­ter­pre­ta­tion was un­ten­able as soon as it was put forth. How­ever, Everett’s ini­tial the­ory was also very un­satis­fy­ing. It only be­came (some­what) at­trac­tive with the much later de­vel­op­ment of de­co­her­ence the­ory, which first made plau­si­ble the claim that no-col­lapse QM evolu­tion could ex­plain our ex­pe­riences. (For most physi­cists who ex­am­ine it se­ri­ously, the claim is still very ques­tion­able).

Hence, the grad­ual in­crease in ac­cep­tance of the MW in­ter­pre­ta­tion is a product both of the old guard dy­ing off and the de­vel­op­ment of bet­ter the­o­ret­i­cal sup­port for MW.

[Damien_R._S.]

I hate that Planck quote. It’s full of “truthi­ness”. I think it is in fact falsified by the his­to­ries of rel­a­tivity, quan­tum me­chan­ics, and con­ti­nen­tal drift/​plate tec­ton­ics. I’m pretty con­fi­dent about the lat­ter, trust­ing Hofs­tadter’s class lec­tures more for the former two.

[Phillip_Huggan]

Per­son­ally, I think the fo­cus here on cog­ni­tive bi­ases in de­ci­sion mak­ing is bi­ased in that it dis­tracts from many other fac­tors (ed­u­ca­tion, info sources, per­son­al­ity, mild men­tal psy­chosis, the level of caf­feine and sugar in one’s blood, etc). If it helps to shed any light on the Pop­per-ian pro­cess of sci­en­tific con­sen­sus, I’ll offer my own anec­dote with the sug­ges­tion that the pro­cess he hy­poth­e­sizes af­fects much more than sci­ence:

I could not be­lieve in 2006 that the Chicago Bears would lose to the Colts. Even though the Colts had pre­vi­ously beaten a scarier aerial at­tack at had a re­vamped defence, I thought the Bears would take it.

What­ever K.Pop­per was de­scribing; I don’t know how true it is, is some sort of vin­dic­tive ego judge­ment call that ex­tends far. Scien­tists are only high­lighted here be­cause they are falsely ex­pected to be ra­tio­nal. In re­al­ity, their re­search is ra­tio­nal, not the pro­cess where they weigh their re­search against the re­search of other sci­en­tists. The lat­ter is con­tam­i­nated by so­ciol­ogy of some sort.

[ME3]

I think that I have only now re­ally un­der­stood what Eliezer has been get­ting at with the past ten or so posts, this idea that you could be a sci­en­tist if you gen­er­ated hy­pothe­ses us­ing a robot con­trol­led Ouija board. I think other read­ers have already said this nu­mer­ous times, but this strikes me as ter­ribly wrong.

First of all, good luck get­ting re­search fund­ing for such hy­pothe­ses (and it wouldn’t be fair to leave out fund­ing from the de­scrip­tion of Science if you’re in­clud­ing in­sti­tu­tional in­er­tia and bias).

And I think we all know that in gen­eral, some­one who used this method would never be able to get any­where in academia, sim­ply be­cause they wouldn’t be re­spected.

That, I think, teaches an im­por­tant les­son. In­di­vi­d­ual sci­en­tists are not re­quired to come up with cor­rect or even plau­si­ble hy­pothe­ses be­cause we all know that in­di­vi­d­ual ra­tio­nal­ity is flawed. But the ag­gre­gate com­mu­nity of sci­en­tists and the peo­ple who fund them work to­gether to eval­u­ate the plau­si­bil­ity of a given hy­poth­e­sis, and thereby effec­tively carry out the Bayesian anal­y­sis that Eliezer speaks of.

So one of many thou­sands of sci­en­tists can pro­pose an ut­terly hare­brained the­ory, and even spend his life on it if he wants, and it will barely reg­ister as a blip on the col­lec­tive sci­en­tific radar. But when SR and GR were pro­posed, it was pretty much taken as a given that they were true, be­cause they HAD to be true. I read some­where that the ex­per­i­ment done by Ed­ding­ton to ver­ify the bend­ing of light around the sun was far from ac­cu­rate enough to ac­tu­ally be a ver­ifi­ca­tion of rel­a­tivity. But it was still taken as a ver­ifi­ca­tion, be­cause ev­ery­one was pretty much con­vinced any­way. And con­versely, no mat­ter how many ex­per­i­ments the cold fu­sion peo­ple do that show some un­ex­pected effects, no­body takes them very se­ri­ously.

Now, you might say that this sys­tem is hor­ribly in­effi­cient, and many peo­ple say this on a reg­u­lar ba­sis. But here, the prob­lem is sim­ply that no in­di­vi­d­ual hu­man be­ing can pro­cess that much in­for­ma­tion, and so the time it takes for a given data point to prop­a­gate through the com­mu­nity is very long. Of course, the in­ter­net helps, and if sci­en­tific jour­nals were free, that would prob­a­bly help also. But ul­ti­mately, I think this in­effi­ciency is pre­cisely the cost of a net­work eval­u­at­ing all of the pri­ors to find out the plau­si­bil­ity of a the­ory.

Of course, it also re­duces a sci­en­tist to noth­ing more than a cog in a ma­chine, and many peo­ple who want to be heroic can’t deal with that. But in real life, no sci­en­tist is ex­pected to eval­u­ate his own hy­poth­e­sis. They are ex­pected to come up with a hy­poth­e­sis, and try to ver­ify it if they can get fund­ing, and let the com­mu­nity de­cide to what ex­tent the re­sults are valid.

[jwflesh]

In real life a real sci­en­tist must test his own hy­poth­e­sis and the hy­pothe­ses of oth­ers. They must de­vise and test a hy­poth­e­sis which lends it­self to spe­cific pre­dic­tions offer­ing a means of test­ing its val­idity. All ob­ser­va­tion, in a spe­cial field of sci­ence, must be ei­ther for or against your hy­poth­e­sis or my hy­poth­e­sis, if the ob­ser­va­tion is to ad­vance sci­ence. Science ad­vances only by in­ves­ti­ga­tors who know how to dis­prove the empty the­o­ries and are already work­ing on it. Science ad­vances only by dis­proofs. It can take many years be­fore the sci­en­tific com­mu­nity gets it.

[Nick_Tarleton]

Sup­pose you came up with some way of gen­er­at­ing hy­pothe­ses that was com­pletely crazy—say, pump­ing a robot-con­trol­led Ouija board with the digits of pi—and the re­sult­ing sug­ges­tions kept on get­ting ver­ified ex­per­i­men­tally. The pure ideal essence of Science wouldn’t skip a beat. The pure ideal essence of Bayes would burst into flames and die.

Why? Me­thinks Bayes would even­tu­ally con­clude there’s some un­ex­pected cor­re­la­tion be­tween re­al­ity and the Ouija board.

(This goes for what ME said as well—if Ouija boards ac­tu­ally gen­er­ated use­ful hy­pothe­ses, even­tu­ally sci­en­tists would wise up and start us­ing them all the time.)

[Caledonian2]

Yes, but a Bayesian Ortho­doxy would never give the Ouija boards a sec­ond look. Science is ex­cel­lent at cor­rect­ing mis­taken be­liefs, but if you let your be­liefs de­ter­mine ex­clu­sively what you’ll look for, you’ll never no­tice the dis­crep­an­cies be­tween your formed be­liefs and the ob­ser­va­tions.

[poke]

I think the prob­lem with all ra­tio­nal­ity-based mod­els of sci­ence is that they don’t take sci­en­tific re­al­ism se­ri­ously enough. That’s not sur­pris­ing given that most of them were de­vel­oped by philoso­phers in re­sponse to rad­i­cal skep­ti­cism about the world and our abil­ity to de­scribe it.

Hy­pothe­ses are con­strained by the world in three ways: (1) the hy­poth­e­sis is, in the first place, con­strained by a set of ini­tial mea­sure­ments and ob­ser­va­tions; (2) the hy­poth­e­sis is con­strained by the tools and skills available for fram­ing and solv­ing the prob­lem, which are a product of pre­vi­ous sci­en­tific de­vel­op­ments, which are in turn a product of phys­i­cal re­al­ity; and (3) the hy­poth­e­sis is con­strained by prior the­ory, which was also sub­ject to con­straints 1-3 dur­ing its de­vel­op­ment, and is thus like­wise con­strained by phys­i­cal re­al­ity. Hy­pothe­ses are also sub­ject to so­ciolog­i­cal con­straints but these are ul­ti­mately grounded in phys­i­cal con­straints. Th­ese are strong con­straints and they ex­ist re­gard­less of whether you in­tro­duce a global nor­ma­tive con­straint of ra­tio­nal­ity (Bayesian or oth­er­wise).

I think any ar­gu­ment for global nor­ma­tive con­straints in sci­ence should first at­tempt to demon­strate that the available phys­i­cal and in­sti­tu­tional con­straints are in­suffi­cient. This needs to be done with refer­ence to sci­ence as prac­ticed rather than other equally ahis­tor­i­cal for­mal mod­els of sci­ence. If we take sci­en­tific re­al­ism se­ri­ously, and be­lieve the ob­jects of sci­ence ex­ist, then they them­selves can ex­plain the suc­cess of sci­ence in ex­plain­ing the world. (Scien­tists are im­plic­itly aware of this. If you ask a sci­en­tist, “How did you reach that con­clu­sion?” they’ll say “I did x, y and z” and list off the prac­ti­cal steps they took. If, how­ever, you ask them, “What is the method­ol­ogy of sci­ence?” they’ll talk about skep­ti­cism or log­i­cal pos­i­tivism or Pop­per’s falsifi­ca­tion­ism or Kuhn’s rev­olu­tions or what­ever hap­pens to be the fad at the time.)

An anal­ogy: If you’re the first to land on the East coast of a new is­land, it’s hardly sur­pris­ing that you and your dece­dents will also be the first to dis­cover its in­land mys­ter­ies as well as the North, South and West coasts; the ge­og­ra­phy of the is­land, as a con­tin­u­ous body of land, en­sures that you can travel from one point to an­other and we need not posit some ad­di­tional nor­ma­tive con­straint that made your peo­ple Great Ex­plor­ers. Like­wise, it’s re­al­ity that de­cides that you get Spe­cial Rel­a­tivity if you turn the wheels on New­to­nian dy­nam­ics enough, and not the alleged ra­tio­nal­ity of the re­searchers in­volved.

[Caledonian2]

One of the pre­con­di­tions of be­com­ing Great Ex­plor­ers is that you have to ex­plore, poke. If you are the first to dis­cover the east­ern coast, but never ven­ture be­yond that, you’ll never find the north­ern, south­ern, or west­ern.

If the re­searchers were not min­i­mally ra­tio­nal, they would never have turned the wheels on New­to­nian physics long enough to dis­cover SR.

[[deleted]]

I would be in­ter­ested in de­vel­op­ing a the­ory of saliency for sci­en­tific hy­pothe­ses. My cur­rent field, com­puter vi­sion, has had some in­ter­est­ing re­sults where saliency can be tar­geted for spe­cific ob­ject types. For ex­am­ple, you could train a “peo­ple spot­ter” and a “bi­cy­cle spot­ter” and then go look at a scene. Both spot­ters will re­port false pos­i­tives, etc., but the spot­ters give you some con­fi­dence (a) about whether the thing you want to find is even in the scene and (b) where in the scene to burn your re­sources when look­ing for it.

I’m not claiming it would be straight­for­ward at all, but a con­ver­sion of this ap­proach aimed at de­tect­ing salient ideas would seem to be the right di­rec­tion. It raises some ques­tions of im­me­di­ate in­ter­est: what sort of fea­ture vec­tor should one ex­tract to quan­tize an idea? There must be ways to de­scribe a set of ex­per­i­ments, say, of max­i­mum mu­tual in­for­ma­tion with re­spect to a set of hy­pothe­ses… that is, you can liter­ally com­pute the ex­per­i­men­tal re­dun­dancy be­tween a set of ex­per­i­ments and whit­tle them down to the ones that bear the max­i­mal rele­vance w.r.t. some set of hy­pothe­ses.

[Nathan Spears]

In be­tween was an in­ter­val where the sci­en­tist ra­tio­nally knew some­thing that the pub­lic so­cial pro­cess of sci­ence hadn’t yet con­firmed.

Isn’t there an­other in­ter­val to con­sider in the pro­cess? There is (of­ten) a point when the sci­en­tist’s in­tu­ition is push­ing him to­ward a hy­poth­e­sis or an in­ter­pre­ta­tion of data which has not yet been con­firmed by his ra­tio­nal­ity. A flash of in­sight is re­quired by the sci­en­tific pro­cess and yet is not ac­counted for.

It seems like a bit of a blind spot—if we had no more flashes of in­sight, the sci­en­tific pro­cess would grind to a halt, no?