3 Levels of Rationality Verification

I strongly sus­pect that there is a pos­si­ble art of ra­tio­nal­ity (at­tain­ing the map that re­flects the ter­ri­tory, choos­ing so as to di­rect re­al­ity into re­gions high in your prefer­ence or­der­ing) which goes be­yond the skills that are stan­dard, and be­yond what any sin­gle prac­ti­tioner singly knows. I have a sense that more is pos­si­ble.

The de­gree to which a group of peo­ple can do any­thing use­ful about this, will de­pend over­whelm­ingly on what meth­ods we can de­vise to ver­ify our many amaz­ing good ideas.

I sug­gest strat­ify­ing ver­ifi­ca­tion meth­ods into 3 lev­els of use­ful­ness:

  • Reputational

  • Experimental

  • Organizational

If your mar­tial arts mas­ter oc­ca­sion­ally fights re­al­is­tic du­els (ideally, real du­els) against the mas­ters of other schools, and wins or at least doesn’t lose too of­ten, then you know that the mas­ter’s rep­u­ta­tion is grounded in re­al­ity; you know that your mas­ter is not a com­plete poseur. The same would go if your school reg­u­larly com­peted against other schools. You’d be keepin’ it real.

Some mar­tial arts fail to com­pete re­al­is­ti­cally enough, and their stu­dents go down in sec­onds against real street­fighters. Other mar­tial arts schools fail to com­pete at all—ex­cept based on charisma and good sto­ries—and their mas­ters de­cide they have chi pow­ers. In this lat­ter class we can also place the splin­tered schools of psy­cho­anal­y­sis.

So even just the ba­sic step of try­ing to ground rep­u­ta­tions in some re­al­is­tic trial other than charisma and good sto­ries, has tremen­dous pos­i­tive effects on a whole field of en­deavor.

But that doesn’t yet get you a sci­ence. A sci­ence re­quires that you be able to test 100 ap­pli­ca­tions of method A against 100 ap­pli­ca­tions of method B and run statis­tics on the re­sults. Ex­per­i­ments have to be repli­ca­ble and repli­cated. This re­quires stan­dard mea­sure­ments that can be run on stu­dents who’ve been taught us­ing ran­domly-as­signed al­ter­na­tive meth­ods, not just re­al­is­tic du­els fought be­tween mas­ters us­ing all of their ac­cu­mu­lated tech­niques and strength.

The field of hap­piness stud­ies was cre­ated, more or less, by re­al­iz­ing that ask­ing peo­ple “On a scale of 1 to 10, how good do you feel right now?” was a mea­sure that statis­ti­cally val­i­dated well against other ideas for mea­sur­ing hap­piness. And this, de­spite all skep­ti­cism, looks like it’s ac­tu­ally a pretty use­ful mea­sure of some things, if you ask 100 peo­ple and av­er­age the re­sults.

But sup­pose you wanted to put hap­pier peo­ple in po­si­tions of power—pay happy peo­ple to train other peo­ple to be hap­pier, or em­ploy the hap­piest at a hedge fund? Then you’re go­ing to need some test that’s harder to game than just ask­ing some­one “How happy are you?”

This ques­tion of ver­ifi­ca­tion meth­ods good enough to build or­ga­ni­za­tions, is a huge prob­lem at all lev­els of mod­ern hu­man so­ciety. If you’re go­ing to use the SAT to con­trol ad­mis­sions to elite col­leges, then can the SAT be defeated by study­ing just for the SAT in a way that ends up not cor­re­lat­ing to other scholas­tic po­ten­tial? If you give col­leges the power to grant de­grees, then do they have an in­cen­tive not to fail peo­ple? (I con­sider it drop-dead ob­vi­ous that the task of ver­ify­ing ac­quired skills and hence the power to grant de­grees should be sep­a­rated from the in­sti­tu­tions that do the teach­ing, but let’s not go into that.) If a hedge fund posts 20% re­turns, are they re­ally that much bet­ter than the in­dices, or are they sel­l­ing puts that will blow up in a down mar­ket?

If you have a ver­ifi­ca­tion method that can be gamed, the whole field adapts to game it, and loses its pur­pose. Col­leges turn into tests of whether you can en­dure the classes. High schools do noth­ing but teach to statewide tests. Hedge funds sell puts to boost their re­turns.

On the other hand—we still man­age to teach en­g­ineers, even though our or­ga­ni­za­tional ver­ifi­ca­tion meth­ods aren’t perfect. So what perfect or im­perfect meth­ods could you use for ver­ify­ing ra­tio­nal­ity skills, that would be at least a lit­tle re­sis­tant to gam­ing?

(Added: Mea­sure­ments with high noise can still be used ex­per­i­men­tally, if you ran­domly as­sign enough sub­jects to have an ex­pec­ta­tion of wash­ing out the var­i­ance. But for the or­ga­ni­za­tional pur­pose of ver­ify­ing par­tic­u­lar in­di­vi­d­u­als, you need low-noise mea­sure­ments.)

So I now put to you the ques­tion—how do you ver­ify ra­tio­nal­ity skills? At any of the three lev­els? Brain­storm, I beg you; even a difficult and ex­pen­sive mea­sure­ment can be­come a gold stan­dard to ver­ify other met­rics. Feel free to email me at sen­tience@pobox.com to sug­gest any mea­sure­ments that are bet­ter off not be­ing pub­li­cly known (though this is of course a ma­jor dis­ad­van­tage of that method). Stupid ideas can sug­gest good ideas, so if you can’t come up with a good idea, come up with a stupid one.

Rep­u­ta­tional, ex­per­i­men­tal, or­ga­ni­za­tional:

  • Some­thing the mas­ters and schools can do to keep it real (re­al­is­ti­cally real);

  • Some­thing you can do to mea­sure each of a hun­dred stu­dents;

  • Some­thing you could use as a test even if peo­ple have an in­cen­tive to game it.

Find­ing good solu­tions at each level de­ter­mines what a whole field of study can be use­ful for—how much it can hope to ac­com­plish. This is one of the Big Im­por­tant Foun­da­tional Ques­tions, so—


(PS: And pon­der on your own be­fore you look at the other com­ments; we need breadth of cov­er­age here.)