Link: Strong Inference

The pa­per “Strong In­fer­ence” by John R. Platt is a meta-anal­y­sis of sci­en­tific method­ol­ogy pub­lished in Science in 1964. It starts off with a won­der­fully ag­gres­sive claim:

Scien­tists these days tend to keep up a po­lite fic­tion that all sci­ence is equal.

The pa­per starts out by ob­serv­ing that some sci­en­tific fields progress much more rapidly than oth­ers. Why should this be?

I think the usual ex­pla­na­tions we tend to think of—such as the tractabil­ity of the sub­ject, or the qual­ity or ed­u­ca­tion of the men drawn into it, or the size of the re­search con­tracts—are im­por­tant but in­ad­e­quate… Rapidly mov­ing fields are fields where a par­tic­u­lar method of do­ing sci­en­tific re­search is sys­tem­at­i­cally used and taught, an ac­cu­mu­la­tive method of in­duc­tive in­fer­ence that is so effec­tive that I think it should be given the name “Strong In­fer­ence”.

The defi­ni­tion of Strong In­fer­ence, ac­cord­ing to Platt, is the for­mal, ex­plicit, and reg­u­lar ad­her­ence to the fol­low­ing pro­ce­dure:

  1. De­vise al­ter­na­tive hy­pothe­ses;

  2. De­vise a cru­cial ex­per­i­ment (or sev­eral of them), with al­ter­na­tive pos­si­ble out­comes, each of which will, as nearly as pos­si­ble, ex­clude one or more of the hy­pothe­ses;

  3. Carry out the ex­per­i­ment so as to get a clean re­sult;

  4. (Goto 1) - Re­cy­cle the pro­ce­dure, mak­ing sub­hy­pothe­ses or se­quen­tial hy­pothe­ses to re­fine the prob­lems that re­main; and so on.

This seems like a sim­ple restate­ment of the sci­en­tific method. Why does Platt bother to tell us some­thing we already know?

The rea­son is that many of us have for­got­ten it. Science is now an ev­ery­day busi­ness. Equip­ment, calcu­la­tions, lec­tures be­come ends in them­selves. How many of us write down our al­ter­na­tives and cru­cial ex­per­i­ments ev­ery day, fo­cus­ing on the ex­clu­sion of a hy­poth­e­sis?

Platt gives us some nice his­tor­i­cal anec­dotes of strong in­fer­ence at work. One is from high-en­ergy physics:

[One of the cru­cial ex­per­i­ments] was thought of one evening at sup­per­time: by mid­night they had ar­ranged the ap­para­tus for it, and by 4am they had picked up the pre­dicted pulses show­ing the non-con­ser­va­tion of par­ity.

The pa­per em­pha­sizes the im­por­tance of sys­tem­at­ic­ity and rigor over raw in­tel­lec­tual fire­power. Roent­gen, pro­ceed­ing sys­tem­at­i­cally, shows us the mean­ing of haste:

Within 8 weeks af­ter the dis­cov­ery of X-rays, Roent­gen had iden­ti­fied 17 of their ma­jor prop­er­ties.

Later, Platt ar­gues against the overuse of math­e­mat­ics:

I think that any­one who asks the ques­tion about sci­en­tific effec­tive­ness will also con­clude that much of the math­e­mat­i­ciz­ing in physics and chem­istry to­day is ir­rele­vant if not mis­lead­ing.

(Fast for­ward to the pre­sent, where we have peo­ple prov­ing the ex­is­tence of Nash equil­ibria in robotics and us­ing Rie­man­nian man­i­folds in com­puter vi­sion, when robots can barely walk up stairs and the prob­lem of face de­tec­tion still has no con­vinc­ing solu­tion.)

One of the ob­sta­cles to hard sci­ence is that hy­pothe­ses must come into con­flict, and one or the other must even­tu­ally win. This cre­ates so­ciolog­i­cal trou­ble, but there’s a solu­tion:

The con­flict and ex­clu­sion of al­ter­na­tives that is nec­es­sary to sharp in­duc­tive in­fer­ence has been all too of­ten a con­flict be­tween men, each with his sin­gle Rul­ing The­ory. But when­ever each man be­gins to have mul­ti­ple work­ing hy­pothe­ses, it be­comes purely a con­flict be­tween ideas.

Fi­nally, Platt sug­gests that all sci­en­tists con­tinu­ally bear in mind The Ques­tion:

But, sir, what ex­per­i­ment could dis­prove your hy­poth­e­sis?


Now, LWers, I am not be­ing rhetor­i­cal, I put these ques­tions to you sincerely: Is ar­tifi­cial in­tel­li­gence, rightly con­sid­ered, an em­piri­cal sci­ence? If not, what is it? Why doesn’t AI make progress like the fields men­tioned in Platt’s pa­per? Why can’t AI re­searchers for­mu­late and test the­o­ries the way high-en­ergy physi­cists do? Can a field which is not an em­piri­cal sci­ence ever make claims about the real world?

If you have time and in­cli­na­tion, try reread­ing my ear­lier post on the Com­pres­sion Rate Method, es­pe­cially the first part, in the light of Platt’s pa­per.

Edited thanks to feed­back from Cupholder.