Confusions Concerning Pre-Rationality

Robin Han­son’s Un­com­mon Pri­ors Re­quire Ori­gin Dis­putes is a short pa­per with, ac­cord­ing to me, a sur­pris­ingly high ra­tio of does-some­thing-in­ter­est­ing-there per char­ac­ter. It is not clearly right, but it mer­its some care­ful con­sid­er­a­tion. If it is right, it offers strong rea­son in sup­port of the com­mon prior as­sump­tion, which is a ma­jor crux of cer­tain mod­est-episte­mol­ogy fla­vored ar­gu­ments.

Wei Dai wrote two posts re­view­ing the con­cepts in the pa­per and dis­cussing prob­lems/​im­pli­ca­tions. I recom­mend re­view­ing those be­fore read­ing the pre­sent post, and pos­si­bly the pa­per it­self as well.

Robin Han­son’s no­tion of pre-ra­tio­nal­ity is: an agent’s coun­ter­fac­tual be­liefs should treat the de­tails of its cre­ation pro­cess like an up­date. If the agent is a Bayesian robot with an ex­plic­itly pro­grammed prior, then the agent’s dis­tri­bu­tion af­ter con­di­tion­ing on any event “pro­gram­mer im­ple­ments prior p” should be ex­actly p.

Th­ese be­liefs are “coun­ter­fac­tual” in that agents are typ­i­cally as­sumed to know their pri­ors already, so that the above con­di­tional prob­a­bil­ity is not well-defined for any choice of p other than the agent’s true prior. This fact leads to a ma­jor com­pli­ca­tion in the pa­per; the pre-ra­tio­nal­ity con­di­tion is in­stead stated in terms of hy­po­thet­i­cal “pre-agents” which have “pre-pri­ors” en­cod­ing the agent’s coun­ter­fac­tual be­liefs about what the world would have been like if the agent had had a differ­ent prior. (I’m cu­ri­ous what hap­pens if we drop that as­sump­tion, so that we can rep­re­sent pre-ra­tio­nal­ity within the agent’s same prior.)

Wei Dai offers an ex­am­ple in which a pro­gram­mer flips a coin to de­ter­mine whether a robot be­lieves coin-flips to have prob­a­bil­ity 2/​3rds or 1/​3rd. Pre-ra­tio­nal­ity seems like an im­plau­si­ble con­straint to put on this robot, be­cause the pro­gram­mer’s coin-flip is not good rea­son to form such ex­pec­ta­tions about other coins.

Wei Dai seems to be ar­gu­ing against a po­si­tion which Robin Han­son isn’t quite ad­vo­cat­ing. Wei Dai’s ac­cu­sa­tion is that pre-ra­tio­nal­ity im­plies a be­lief that the pro­cess which cre­ated you was it­self a ra­tio­nal pro­cess, which is not always plau­si­ble. In­deed, it’s easy to see this in­ter­pre­ta­tion in the math. How­ever, Robin Han­son’s re­sponse in­di­cates that he doesn’t see it:

I just don’t see pre-ra­tio­nal­ity be­ing much tied to whether you in fact had a ra­tio­nal cre­ator. The point is, as you say, to con­sider the info in the way you were cre­ated.

Un­for­tu­nately, the dis­cus­sion in the com­ments doesn’t go any fur­ther on this point. How­ever, we can make some in­fer­ences about Robin Han­son’s po­si­tion from the pa­per it­self.

Robin Han­son does not dis­cuss the robot/​pro­gram­mer ex­am­ple; in­stead, he dis­cusses the pos­si­bil­ity that peo­ple have differ­ing pri­ors due to ge­netic fac­tors. Far from claiming peo­ple are obli­gated by ra­tio­nal­ity prin­ci­ples to treat in­her­ited pri­ors as ra­tio­nal, Robin Han­son says that be­cause we know some ran­dom­ness is in­volved in Men­delian in­her­i­tance, we can’t both rec­og­nize the ar­bi­trari­ness of our prior’s ori­gin and stick with that prior. Quot­ing the pa­per on this point:

Men­del’s rules of ge­netic in­her­i­tance, how­ever, are sym­met­ric and ran­dom be­tween siblings. If op­ti­mism were coded in genes, you would not ac­quire an op­ti­mism gene in situ­a­tions where op­ti­mism was more ap­pro­pri­ate, nor would your sister’s at­ti­tude gene track truth any worse than your at­ti­tude gene does.
Thus it seems to be a vi­o­la­tion of pre-ra­tio­nal­ity to, con­di­tional on ac­cept­ing Men­del’s rules, al­low one’s prior to de­pend on in­di­vi­d­ual vari­a­tions in ge­net­i­cally-en­coded at­ti­tudes. Hav­ing your prior de­pend on species-av­er­age ge­netic at­ti­tudes may not vi­o­late pre-ra­tio­nal­ity, but this would not jus­tify differ­ing pri­ors within a species.

Robin Han­son sug­gests that pre-ra­tio­nal­ity is only plau­si­ble con­di­tional on some knowl­edge we have gained through­out our life­time about our own ori­gins. He posits a sen­tence B which con­tains this knowl­edge, and sug­gests that the pre-ra­tio­nal­ity con­di­tion can be rel­a­tivized to B. In the above-quoted case, B would con­sist of Men­delian in­her­i­tance and the ge­net­ics of op­ti­mism. Robin Han­son is not say­ing that ge­netic in­her­i­tance of op­ti­mism or pes­simism is a ra­tio­nal pro­cess, but rather, he is say­ing that once we know about these ge­netic fac­tors, we should ad­just our pes­simism or op­ti­mism to­ward the species av­er­age. After perform­ing this ad­just­ment, we are pre-ra­tio­nal: we con­sider any re­main­ing in­fluences on our prob­a­bil­ity dis­tri­bu­tion to have been ra­tio­nal.

Wei Dai’s ar­gu­ment might be char­i­ta­bly in­ter­preted as ob­ject­ing to this po­si­tion by offer­ing a con­crete case in which a ra­tio­nal agent does not up­date to pre-ra­tio­nal­ity in this way: the robot has no mo­ti­va­tion to ad­just for the ran­dom noise in its prior, de­spite its recog­ni­tion of the ir­ra­tional­ity of the pro­cess by which it in­her­ited this prior. How­ever, I agree with Robin Han­son that this is in­tu­itively quite prob­le­matic, even if no laws of prob­a­bil­ity are vi­o­lated. There is some­thing wrong with the robot’s po­si­tion, even if the robot lacks cog­ni­tive tools to es­cape this epistemic state.

How­ever, Wei Dai does offer a sig­nifi­cant re­sponse to this: he com­plains that Robin Han­son says too lit­tle about what the robot should do to be­come pre-ra­tio­nal from its flawed state. The pre-ra­tio­nal­ity con­di­tion pro­vides no guidance for the robot. As such, what guidance can pre-ra­tio­nal­ity offer to hu­mans? Robin Han­son’s pa­per ad­mits that we have to con­di­tion on B to be­come pre-ra­tio­nal, but offers no ac­count what­so­ever about the struc­ture of this up­date. What nor­ma­tive struc­ture should we re­quire of pri­ors so that an agent be­comes pre-ra­tio­nal when con­di­tioned on the ap­pro­pri­ate B?

Here is the text of Wei Dai’s sage com­plaint:

As­sum­ing that we do want to be pre-ra­tio­nal, how do we move from our cur­rent non-pre-ra­tio­nal state to a pre-ra­tio­nal one? This is some­what similar to the ques­tion of how do we move from our cur­rent non-ra­tio­nal (ac­cord­ing to or­di­nary ra­tio­nal­ity) state to a ra­tio­nal one. Ex­pected util­ity the­ory says that we should act as if we are max­i­miz­ing ex­pected util­ity, but it doesn’t say what we should do if we find our­selves lack­ing a prior and a util­ity func­tion (i.e., if our ac­tual prefer­ences can­not be rep­re­sented as max­i­miz­ing ex­pected util­ity).
The fact that we don’t have good an­swers for these ques­tions per­haps shouldn’t be con­sid­ered fatal to pre-ra­tio­nal­ity and ra­tio­nal­ity, but it’s trou­bling that lit­tle at­ten­tion has been paid to them, rel­a­tive to defin­ing pre-ra­tio­nal­ity and ra­tio­nal­ity. (Why are ra­tio­nal­ity re­searchers more in­ter­ested in know­ing what ra­tio­nal­ity is, and less in­ter­ested in know­ing how to be ra­tio­nal? Also, BTW, why are there so few ra­tio­nal­ity re­searchers? Why aren’t there hordes of peo­ple in­ter­ested in these is­sues?)

I find my­self in the some­what awk­ward po­si­tion of agree­ing strongly with Robin Han­son’s in­tu­itions here, but also hav­ing no idea how it should work. For ex­am­ple, sup­pose that we have a robot whose prob­a­bil­is­tic be­liefs are oc­ca­sion­ally mod­ified by cos­mic rays. Th­ese mod­ifi­ca­tion events can be thought of as the en­vi­ron­ment writ­ing a new “prior” into the agent. We can­not perfectly safe­guard the agent against this, but we can write the agent’s prob­a­bil­ity dis­tri­bu­tion such that so long as it is not too dam­aged, it can self-re­pair when it sees ev­i­dence that its be­liefs have been mod­ified by the en­vi­ron­ment. This seems like an up­dat­ing-to-pre-ra­tio­nal­ity move, with “a cos­mic ray hit you in this mem­ory cell” play­ing the role of B.

Similarly, it seems rea­son­able to do some­thing like av­er­age be­liefs with some­one if you dis­cover that your differ­ing be­liefs are due only to ge­netic chance. Yet, it does not seem similarly rea­son­able to av­er­age val­ues, de­spite the dis­tinc­tion be­tween be­liefs and prefer­ences be­ing some­what fuzzy.

This is made even more awk­ward by the fact that Robin Han­son has to cre­ate the whole pre-prior frame­work in or­der to state his new ra­tio­nal­ity con­straint.

The idea seems to be that a pre-prior is not a be­lief struc­ture which an ac­tual agent has, but rather, is a kind of plau­si­ble ex­trap­o­la­tion of an agent’s be­lief struc­ture which we layer on top of the true be­lief struc­ture in or­der to rea­son about the new ra­tio­nal­ity con­straint. If so, how could this kind of ra­tio­nal­ity con­straint be com­pel­ling to an agent? The agent it­self doesn’t have any pre-prior. Yet, if we have an in­tu­ition that Robin Han­son’s ar­gu­ment im­plies some­thing about hu­mans, then we our­selves are agents who find ar­gu­ments in­volv­ing pre-pri­ors to be rele­vant.

Alter­na­tively, pre-pri­ors could be cap­tur­ing in­for­ma­tion about coun­ter­fac­tual be­liefs which the agent it­self has. This seems less ob­jec­tion­able, but it brings in tricky is­sues of coun­ter­fac­tual rea­son­ing. I don’t think this is likely to be the right path to prop­erly for­mal­iz­ing what is go­ing on, ei­ther.

I see two clusters of ap­proaches:

  • What ra­tio­nal­ity con­di­tions might we im­pose on a Bayesian agent such that it up­dates to pre-ra­tio­nal­ity given “ap­pro­pri­ate” B? Can we for­mal­ize this purely within the agent’s own prior, with­out the use of pre-pri­ors?

  • What can we say about agents be­com­ing ra­tio­nal from ir­ra­tional po­si­tions? What should agents do when they no­tice Dutch Books against their be­liefs, or money-pumps against their prefer­ences? (Log­i­cal In­duc­tion is a some­what helpful story about the former, but not the lat­ter.) Can we char­ac­ter­ize the re­ceiver of de­ci­sion-the­o­retic ar­gu­ments such as the VNM the­o­rem, who would find such ar­gu­ments in­ter­est­ing? If we can pro­duce any­thing in this di­rec­tion, can it say any­thing about Robin Han­son’s ar­gu­ments con­cern­ing pre-ra­tio­nal­ity? Does it give a model, or can it be mod­ified to give a model, of up­dat­ing to pre-ra­tionlity?

It seems to me that there is some­thing in­ter­est­ing go­ing on here, and I wish that there were more work on Han­so­nian pre-ra­tio­nal­ity and Wei Dai’s ob­jec­tion.