[Question] What are the open problems in Human Rationality?

LessWrong has been around for 10+ years, CFAR’s been at work for around 6, and I think there have been at least a few other groups or in­di­vi­d­u­als work­ing on what I think of as the “Hu­man Ra­tion­al­ity Pro­ject.”

I’m in­ter­ested, es­pe­cially from peo­ple who have in­vested sig­nifi­cant time in at­tempt­ing to push the ra­tio­nal­ity pro­ject for­ward, what they con­sider the ma­jor open ques­tions fac­ing the field. (More de­tails in this com­ment)

“What is the Ra­tion­al­ity Pro­ject?”

I’d pre­fer to leave “Ra­tion­al­ity Pro­ject” some­what vague, but I’d roughly sum­ma­rize it as “the study of how to have op­ti­mal be­liefs and make op­ti­mal de­ci­sions while run­ning on hu­man wet­ware.”

If you have your own sense of what this means or should mean, feel free to use that in your an­swer. But some bits of con­text for a few pos­si­ble av­enues you could in­ter­pret this through:

Early LessWrong fo­cused a lot of cog­ni­tive bi­ases and how to ac­count for them, as well as Bayesian episte­mol­ogy.

CFAR (to my knowl­edge, roughly) started from a similar van­tage point and even­tu­ally started mov­ing in the di­rec­tion of “how to do you figure out what you ac­tu­ally want and bring your­self into ‘in­ter­nal al­ign­ment’ when you want mul­ti­ple things, and/​or differ­ent parts of you want differ­ent things and are work­ing at cross pur­poses. It also looked a lot into Dou­ble Crux, as a tool to help peo­ple dis­agree more pro­duc­tively.

CFAR and Lev­er­age both ended up ex­plor­ing in­tro­spec­tion as a tool.

Fore­cast­ing as a field has ma­tured a bit. We have the Good Judg­ment pro­ject.

Be­hav­ioral Eco­nomics has be­gun to de­velop as a field.

I re­cently read “How to Mea­sure Any­thing”, and was some­what struck at how it tack­led pre­dic­tion, cal­ibra­tion and de­ter­min­ing key un­cer­tain­ties in a fairly rigor­ous, pro­fes­sion­al­ized fash­ion. I could imag­ine an al­ter­nate his­tory of LessWrong that had em­pha­sized this more strongly.

With this vague con­stel­la­tion of or­ga­ni­za­tions and re­search ar­eas, ges­tur­ing at an over­all field…

...what are the big open ques­tions the field of Hu­man Ra­tion­al­ity needs to an­swer, in or­der to help peo­ple have more ac­cu­rate be­liefs and/​or make bet­ter de­ci­sions?