Why CFAR’s Mission?

Re­lated to:

Briefly put, CFAR’s mis­sion is to im­prove the san­ity/​think­ing skill of those who are most likely to ac­tu­ally use­fully im­pact the world.
I’d like to ex­plain what this mis­sion means to me, and why I think a high-qual­ity effort of this sort is es­sen­tial, pos­si­ble, and ur­gent.
I used a Q&A for­mat (with imag­i­nary Q’s) to keep things read­able; I would also be very glad to Skype 1-on-1 if you’d like some­thing about CFAR to make sense, as would Pete Michaud. You can sched­ule a con­ver­sa­tion au­to­mat­i­cally with me or Pete.


Q: Why not fo­cus ex­clu­sively on spread­ing al­tru­ism? Or else on “rais­ing aware­ness” for some par­tic­u­lar known cause?

Briefly put: be­cause his­tor­i­cal roads to hell have been pow­ered in part by good in­ten­tions; be­cause the con­tem­po­rary world seems bot­tle­necked by its abil­ity to figure out what to do and how to do it (i.e. by ideas/​cre­ativity/​ca­pac­ity) more than by folks’ will­ing­ness to sac­ri­fice; and be­cause ra­tio­nal­ity skill and epistemic hy­giene seem like skills that may dis­t­in­guish ac­tu­ally use­ful ideas from in­effec­tive or harm­ful ones in a way that “good in­ten­tions” can­not.

Q: Even given the above—why fo­cus ex­tra on san­ity, or true be­liefs? Why not fo­cus in­stead on, say, com­pe­tence/​use­ful­ness as the key de­ter­mi­nant of how much do-good­ing im­pact a mo­ti­vated per­son can have? (Also, have you ever met a Less Wronger? I hear they are an­noy­ing and have lots of prob­lems with “akra­sia”, even while prid­ing them­selves on their high “epistemic” skills; and I know lots of peo­ple who seem “less ra­tio­nal” than Less Wrongers on some axes who would nev­er­the­less be more use­ful in many jobs; is this “epistemic ra­tio­nal­ity” thingy ac­tu­ally the thing we need for this world-im­pact thingy?...)

This is an in­ter­est­ing one, IMO.

Ba­si­cally, it seems to me that epistemic ra­tio­nal­ity, and skills for form­ing ac­cu­rate ex­plicit world-mod­els, be­come more use­ful the more am­bi­tious and con­fus­ing a prob­lem one is tack­ling.

For ex­am­ple:

If I have one floor to sweep, it would be best to hire a per­son who has pre-ex­ist­ing skill at sweep­ing floors.

If I have 250 floors to sweep, it would be best to have some­one en­er­getic and per­cep­tive, who will stick to the task, no­tice whether they are suc­ceed­ing, and im­prove their effi­ciency over time. An “all round com­pe­tent hu­man be­ing”, maybe.

If I have 10^25 floors to sweep, it would… be rather difficult to win at all, ac­tu­ally. But if I can win, it prob­a­bly isn’t by har­ness­ing my pre-ex­ist­ing skill at floor-sweep­ing, nor even (I’d claim) my pre-ex­ist­ing skill at “gen­eral hu­man com­pe­tence”. It’s prob­a­bly by us­ing the foun­da­tions of sci­ence and/​or poli­tics to (some­how) cre­ate some to­tally crazy method of get­ting the floors swept (a pro­cess that would prob­a­bly re­quire ac­tu­ally ac­cu­rate be­liefs, and thus epistemic ra­tio­nal­ity).

The world’s most im­por­tant prob­lems look to me more like that third ex­am­ple. And, again, it seems to me that to solve prob­lems of that sort—to iter­ate through many wrong guesses and some­how piece to­gether an ac­cu­rate model un­til one finds a work­able path­way for do­ing what origi­nally looked im­pos­si­ble—with­out get­ting stuck in dead ends or wrong turns or in­born or so­cietal prej­u­dices—it is damn helpful to have some­thing like epistemic ra­tio­nal­ity. (Com­pe­tence is pretty darn helpful too—it’s good to e.g. be able to go out there and get data; to be able to form net­work­ing re­la­tions with folks who already know things; etc. -- but epistemic ra­tio­nal­ity is nec­es­sary in a more fun­da­men­tal way.)

For the sake of con­crete­ness, I will claim that AI-re­lated ex­is­ten­tial risk is among hu­man­ity’s most im­por­tant prob­lems, and that it is damn con­fus­ing, damn hard, and re­ally re­ally needs some­thing like epistemic ra­tio­nal­ity and not just some­thing like al­tru­ism and com­pe­tence if one is to im­pact it pos­i­tively, rather than just, say, ran­domly im­pact­ing it. I’d be glad to dis­cuss in the com­ments.

Q: Why sup­pose “san­ity skill” can be in­creased?

Let’s start with an eas­ier ques­tion: why sup­pose think­ing skills (of any sort) can be in­creased?

The an­swer to that one is easy: Be­cause we see it done all the time.

The math stu­dent who ar­rives at col­lege and does math for the first time with oth­ers is ab­sorb­ing a kind of think­ing skill; thus math­e­mat­i­ci­ans dis­cuss a per­son’s “math­e­mat­i­cal ma­tu­rity”, as a prop­erty dis­tinct from (al­though re­lated to) their learn­ing of this and that math the­o­rem.

Similarly, the coder who hacks her way through a bunch of soft­ware pro­jects and learns sev­eral pro­gram­ming lan­guages will have a much eas­ier time learn­ing her 8th lan­guage than she did her first; ba­si­cally be­cause, some­where along the line, she learned to “think like a com­puter sci­en­tist”...

The claim that “san­ity skill” is a type of think­ing skill and that it can be in­creased is some­what less ob­vi­ous. I am per­son­ally con­vinced that the LW Se­quences /​ AI to Zom­bies gave me some­thing, and gave some­thing similar to oth­ers I know, and that hang­ing out in per­son with Eliezer Yud­kowsky, Michael Vas­sar, Carl Shul­man, Nick Bostrom, and oth­ers gave me more of that same thing; a “same thing” that in­cluded e.g. ac­tu­ally try­ing to figure it out, mak­ing be­liefs pay rent in an­ti­ci­pated ex­pe­rience; us­ing ar­ith­metic to en­tan­gle differ­ent pieces of my be­liefs; and so on.

I similarly have the strong im­pres­sion that e.g. Feyn­man’s and Munger’s pop­u­lar writ­ings of­ten pass on pieces of this same thing; that the con­ver­gence be­tween the LW Se­quences and Tet­lock’s Su­perfore­cast­ing train­ing is non-co­in­ci­den­tal; that the con­ver­gence be­tween CFAR’s work­shop con­tents and a typ­i­cal MBA pro­gram’s con­tents is non-co­in­ci­den­tal (though we were un­aware of it when cre­at­ing our ini­tial draft); and more gen­er­ally that there are many types of think­ing skill that are rou­tinely learned/​taught and that non-triv­ially aid the pro­cess of com­ing to ac­cu­rate be­liefs in tricky do­mains. I up­date to­ward this partly from the above con­ver­gences; from the fact that Tet­lock’s train­ing seems to work; from the fact that e.g. Feyn­man and Munger (and for that mat­ter Thiel, Ray Dalio, Fran­cis Ba­con, and a num­ber of oth­ers) were shock­ingly con­ven­tion­ally suc­cess­ful and ad­vo­cated similar things; and from the fact that there is quite a bit of “san­ity” ad­vice that is ob­vi­ously cor­rect once stated, but that we don’t au­to­mat­i­cally do (ad­vice like “bother to look at the data; and try to up­date if the data doesn’t match your pre­dic­tions”).

So, yes, I sus­pect that there is some por­tion of san­ity that can some­times be learned and taught. And I sus­pect this por­tion can be in­creased fur­ther with work.

Q. Even if you can train skills: Why go through all the trou­ble and com­pli­ca­tions of try­ing to do this, rather than try­ing to find and re­cruit peo­ple who already have the skills?

The short ver­sion: be­cause there don’t seem to be enough peo­ple out there with the rele­vant skills (yet), and be­cause it does seem to be pos­si­ble to help peo­ple in­crease their skills with train­ing.
How can I tell there aren’t enough peo­ple out there, in­stead of sup­pos­ing that we haven’t yet figured out how to find and re­cruit them?
Ba­si­cally, be­cause it seems to me that if peo­ple had re­ally huge amounts of epistemic ra­tio­nal­ity + com­pe­tence + car­ing, they would already be im­pact­ing these prob­lems. Their huge amounts of epistemic ra­tio­nal­ity and com­pe­tence would al­low them to find a path to high im­pact; and their car­ing would com­pel them to do it.
I re­al­ize this is a some­what odd thing to say, and may not seem true to most read­ers. It didn’t used to seem true to me. I my­self found the idea of ex­is­ten­tial risk partly through luck, and so, be­ing un­able to pic­ture thinkers who had skills I lacked, I felt that any­one would need luck to find out about ex­is­ten­tial risk. Re­cruiters, I figured, could fix that luck-gap by find­ing peo­ple and tel­ling them about the risks.
How­ever, there are folks who can rea­son their own way to the con­cept. And there are folks who, hav­ing no­ticed how much these is­sues mat­ter, can lo­cate novel paths that may plau­si­bly im­pact hu­man out­comes. And even if there weren’t folks of this sort, there would be hy­po­thet­i­cal peo­ple of this sort; much of what is luck at some skill-lev­els can in prin­ci­ple be made into skill. So, find­ing that only a limited num­ber of peo­ple are work­ing effec­tively on these prob­lems (pro­vided one is right about that as­sess­ment) does offer an up­per bound on how many peo­ple to­day can ex­ceed a cer­tain com­bined level of epistemic ra­tio­nal­ity, com­pe­tence, and car­ing about the world.
And of course, we should also work to re­cruit com­pe­tence and epistemic skill and car­ing wher­ever we can find them. (To help such folk find their high­est im­pact path, what­ever that turns out to be.) We just shouldn’t ex­pect such re­cruit­ing to be enough, and shouldn’t stop there. Espe­cially given the stakes. And given the things that may hap­pen if e.g. poli­ti­cal com­pe­tence, “raised aware­ness”, and re­cruited good in­ten­tions are al­lowed out­pace in­sight and good epistemic norms on e.g. AI-re­lated ex­is­ten­tial risk, or other similarly con­fus­ing is­sues. An­drew Critch makes this point well with re­spect to AI safety in a re­cent post; I agree with ba­si­cally ev­ery­thing he says there.
Q. Why do you some­times talk of CFAR’s mis­sion as “im­prov­ing think­ing skill”, and other times talk of im­prov­ing all of epistemic ra­tio­nal­ity, com­pe­tence, and do-good­ing? Which one are you af­ter?

The main goal is think­ing skill. Speci­fi­cally, think­ing skill among those most likely to suc­cess­fully use it to pos­i­tively im­pact the world.

Com­pe­tence and car­ing are rele­vant sec­ondary goals: some of us have a con­jec­ture that deep epistemic ra­tio­nal­ity can be use­ful for cre­at­ing com­pe­tence and car­ing, and of course com­pe­tence and car­ing about the world are also di­rectly use­ful for im­pact­ing the world’s prob­lems. But CFAR wants to in­crease com­pe­tence and car­ing via teach­ing rele­vant pieces of think­ing skill, and not via spe­cial-case hacks. For ex­am­ple, we want to help peo­ple stay tuned into what they care about even when this is painful, and to help peo­ple no­tice their aver­sions and sort through which of their aver­sions are and aren’t based in ac­cu­rate im­plicit mod­els. We do not want to use ran­dom emo­tional ap­peals to boost spe­cific cause ar­eas, nor to use other spe­cial-case hacks that hap­pen to boost effi­cacy in a man­ner opaque to par­ti­ci­pants.

Why fo­cus pri­mar­ily on think­ing skill? Partly so we can have fo­cus enough as an or­ga­ni­za­tion so as to ac­tu­ally do any­thing at all. (Or­ga­ni­za­tions that try to ac­com­plish sev­eral things at once risk ac­com­plish­ing none—and “epistemic ra­tio­nal­ity” is more of a sin­gle thing.) Partly so our work­shop par­ti­ci­pants and other learn­ers can similarly have fo­cus as learn­ers. And partly be­cause, as dis­cussed above, it is very very hard to in­ter­vene in global af­fairs in such a way as to ac­tu­ally have pos­i­tive out­comes, and not merely out­comes one pre­tends will be pos­i­tive; and fo­cus­ing on ac­tual think­ing skill seems like a bet­ter bet for prob­lems as con­fus­ing as e.g. ex­is­ten­tial risk.

Why in­clude com­pe­tence and car­ing at all, then? Be­cause high-perform­ing hu­mans make use of large por­tions of their minds (I think), and if we fo­cus only on “ac­cu­rate be­liefs” in a nar­row sense (e.g., do­ing analogs of Tet­locks fore­cast­ing train­ing and noth­ing else), we are apt to gen­er­ate “straw less­wrongers” whose “ra­tio­nal­ity” ap­plies mainly to their ex­plicit be­liefs… peo­ple who can nit­pick in­cor­rect state­ments and can in this way at­tempt ac­cu­rate ver­bal state­ments, but who are not cre­atively gen­er­a­tive, do not have the iter­ated en­ergy/​com­pe­tence/​rapid iter­a­tion re­quired to launch a startup, and can­not run good fast re­al­time so­cial skills. We aim to do bet­ter. And we sus­pect that work­ing to hit com­pe­tence and car­ing via what one might call “deep epistemic ra­tio­nal­ity” is a route in.

Q. Can a small or­ga­ni­za­tion re­al­is­ti­cally do all that with­out los­ing Po­modoro virtue? (By “Po­modoro virtue”, I mean the abil­ity to fo­cus on one thing at a time and so to ac­tu­ally make progress, in­stead of los­ing one­self amidst the dis­trac­tion of 20 goals.)

We think so, and we think the new core/​labs di­vi­sion within CFAR will help. Ba­si­cally, Core will be work­ing to scale up the work­shops and re­lated in­fras­truc­ture, which should give a nice track­able set of num­bers to op­ti­mize—num­bers that, if grown, will en­able bet­ter fi­nan­cial health for CFAR and will also en­able a much larger set of peo­ple to train in ra­tio­nal­ity.

Labs will be fo­cus­ing on im­pact­ing smaller num­bers of peo­ple who are poised to im­pact ex­is­ten­tial risk (mainly), and on see­ing whether our “im­pacts” on these folk do in fact seem to help with their im­pact on the world.

We will con­tinue to work to­gether on many pro­jects and to trade ideas fre­quently, but I sus­pect that this sep­a­ra­tion into two goals will give more “po­morodo virtue” to the whole or­ga­ni­za­tion.

Q. What is CFAR’s re­la­tion­ship to ex­is­ten­tial risk? And what should it be?

CFAR’s mis­sion is to im­prove the san­ity/​think­ing skill of those who are most likely to ac­tu­ally use­fully im­pact the world—via what­ever cause ar­eas may be most im­por­tant. Many of us sus­pect that AI-re­lated ex­is­ten­tial risk is an area with huge po­ten­tial for use­ful im­pact; and so we are fo­cus­ing partly on helping meet tal­ent gaps in that field. This fo­cus also gives us more “po­modoro virtue”—it is eas­ier to track whether e.g. the MIRI Sum­mer Fel­lows Pro­gram helped boost re­search on AI safety, than it is to track whether a work­shop had “good im­pacts on the world” in some more gen­eral sense.

It is im­por­tant to us that the fo­cus re­main on “high im­pact path­ways, what­ever those turn out to be”, that we do not pro­pa­gan­dize for par­tic­u­lar pre-set an­swers (rather, that we as­sist folks in think­ing things through in an un­hin­dered way), and that we work to­ward a kind of think­ing skill that may let peo­ple bet­ter as­sess what paths are ac­tu­ally high im­pact for hav­ing pos­i­tive effects in the world, and to over­come flaws in our cur­rent think­ing.

Q. Should I do “Earn­ing to Give”? Also: I heard that there are big fun­ders around now and so “earn­ing to give” is no longer a sen­si­ble thing for most peo­ple to do; is that true? And what does all this have to do with CFAR?

IMO, earn­ing to give re­mains a path­way for cre­at­ing very large pos­i­tive im­pacts on the world.
For ex­am­ple, CFAR, MIRI, and many of the or­ga­ni­za­tions un­der the CEA um­brella seem to me to be both high po­ten­tial im­pact, and sub­stan­tially fund­ing limited right now, such that fur­ther dona­tion is likely to cause a sub­stan­tial in­crease in how much good these or­ga­ni­za­tions ac­com­plish.
This is an in­ter­est­ing claim to make in a world where e.g. Good Ven­tures, Elon Musk, and oth­ers are already putting very large sums of money into mak­ing the world bet­ter; if you ex­pect to make a large differ­ence via in­di­vi­d­ual dona­tion, you must im­plic­itly ex­pect that you can pick dona­tion tar­gets for your own small­ish sum bet­ter than they could, at least at the mar­gin. (One fac­tor that makes this more plau­si­ble is that you can af­ford to put in far more time/​at­ten­tion/​thought per dol­lar than they can.)
Alter­nately stated: the world seems to me to be short, not so much on money, as on un­der­stand­ing of what to do with money; and an “Earn­ing to Give” ca­reer seems to me to po­ten­tially make sense in­so­far as it is a de­ci­sion to re­ally ac­tu­ally try to figure out how to get hu­man­ity to a win-state (or how to oth­er­wise do what­ever it is that is ac­tu­ally worth­while), es­pe­cially in­so­far as you seem to see low-hang­ing fruit for high-im­pact dona­tion. Ar­gu­ing with varied oth­ers who have thought about it is per­haps the fastest route to­ward a bet­ter-de­vel­oped in­side view, to­gether with set­ting a 5 minute timer and at­tempt­ing to solve it your­self. “Earn­ing to give”ers, IMO, con­tribute in pro­por­tion to the amount of non-stan­dard epistemic skill they de­velop, and not just in pro­por­tion to their giv­ing amount; and, much as with Giv­ing Games, know­ing that you mat­ter to­day can provide a rea­son to de­velop a wor­ld­view for real.
This has to do with CFAR both be­cause I ex­pect us to ac­com­plish far more good if we have money to do it, and be­cause folks ac­tu­ally try­ing to figure out how to im­pact the world are build­ing the kind of think­ing skill CFAR is all about.