CEA does not seem to be credibly high impact

I am highly grate­ful to Alexey Morgunov and Adam Casey for re­view­ing and com­ment­ing on an ear­lier draft of this post, and pes­ter­ing me into mi­grat­ing the con­tent from many emails to a some­what co­her­ent post.

Will Crouch has posted about the Cen­tre for Effec­tive Altru­ism and in a fol­low up post dis­cussed ques­tions in more de­tail. The gen­eral sense of the dis­cus­sion of that post was that the ar­gu­ments were con­vinc­ing and that donat­ing to CEA is a good idea. Re­cently, he vis­ited Cam­bridge, pri­mar­ily to dis­cuss 80,000 hours, and sev­eral Cam­bridge LWers spoke with him. Th­ese dis­cus­sions caused a num­ber of us to sub­stan­tially down­grade our es­ti­mates of the effec­tive­ness of CEA, and made our con­cerns more con­crete.

We’re aware that our kind of­ten don’t co­op­er­ate well, but we are con­cerned that at pre­sent CEA’s pro­jects are un­likely to cash out into large num­bers of peo­ple chang­ing their be­havi­our. Ul­ti­mately, we are con­cerned that the space for high im­pact meta-char­ity is limited, and that if CEA is sub­op­ti­mal this will have large op­por­tu­nity costs. We want CEA to change the world, and would pre­fer this hap­pens quickly.

The key ar­gu­ment in favour of donat­ing money to CEA which was pre­sented by Will was that by donat­ing $1 to CEA you pro­duce more than $1 in dona­tions to the most effec­tive char­i­ties. We pre­sent some ap­par­ent difficul­ties with this re­main­ing true on the mar­gin. We also pre­sent more gen­eral wor­ries with CEA as an or­gani­sa­tion un­der these head­ings:

Trans­parency
Cost effec­tive­ness es­ti­mates
Re­search
80,000 hours
Im­pact of 80,000 hours ad­vice
Con­tent of 80,000 hours ad­vice
The 80,000 hours pledge
S­cope and Goals
Speed of growth
Ambition

Transparency

It is wor­ry­ing how lit­tle of the key in­for­ma­tion about CEA is pub­li­cly available. This makes as­sess­ment hard. By con­trast to GiveWell, CEA pro­grams are not par­tic­u­larly open about where their money is spent, what their marginal goals are, or what they are do­ing in­ter­nally. As pre­sented on­line, the ma­jor­ity of both 80,000 hours’ and GWWC’s day to day ac­tivity is main­tain­ing blogs. Th­ese blogs are not sub­stan­tial by com­par­i­son to, say, OB in terms of their fre­quency of con­tent or their fre­quency of in­sight. Con­cretely, it does not seem that CEA is be­ing tran­par­ent in the sense of GiveWell.

Qn: How does CEA think its pro­grams would score on a GiveWell as­sess­ment?
Qn: Does CEA think that GiveWell’s as­sess­ments sys­tem­i­cally go wrong?

Qn: Does CEA con­sider the blogs to be a sub­stan­tial source of im­pact? What ex­ter­nal as­sess­ments or ob­jec­tive data sup­port a claim of im­pact from the blogs?

Cost effec­tive­ness estimates

As pre­sented on­line and in per­son, CEA does not pre­sent as hav­ing cred­ible mod­els for their fu­ture im­pact. The GWWC site, for ex­am­ple, claims that from 291 mem­bers there will be £72.68M pledged. This equates to £250K /​ per­son over the course of their life. Claiming that this level of pledg­ing will oc­cur re­quires ei­ther un­rea­son­able rates of dona­tion or multi-decade pay­ment sched­ules. If, in line with GWWC’s pro­jec­tions, around 50% of peo­ple will main­tain their dona­tions, then as­sum­ing a lin­ear drop off the ex­pected pledge from a full time mem­ber is around £375K. Over a life­time, this is es­sen­tially £10K /​ year. It seems im­plau­si­ble that ex­pected mean an­nual earn­ings for GWWC mem­bers is of or­der £100K.

Qn: On what ba­sis does GWWC as­sert that its near 300 mem­bers are cred­ibly pre­com­mit­ted to donat­ing £72.68M?

Look­ing at valu­ing marginal im­pacts, it would be hoped CEA’s pro­grams are bet­ter. For ex­am­ple, it has been stated that GWWC has an in­ter­nal price of around £1700 for new pledges. This does not ap­pear to ex­tend to new pro­grams, or to por­tions of 80,000 hours. In re­cent con­ver­sa­tion with Will Crouch, he was asked what marginal value was placed on hav­ing a new in­tern in Cam­bridge (UK). There was no nu­mer­ate re­sponse. In­deed, the as­sorted es­ti­mates do not co­here. If new pledges are worth £10K /​ year in ex­pec­ta­tion, and even 10% of the dona­tions flow into CEA, then an in­tern gen­er­at­ing 20 marginal pledges is a win­ning propo­si­tion for CEA at their stated wage level. If the hori­zon time for 20 pledges from one worker is larger than CEA can af­ford to wait, then it is not clear that CEA has an effec­tive pro­gram for us­ing their in­terns.

Qn: What does GWWC or 80,000 hours see as the marginal im­pact of one ad­di­tional grad stu­dent in full time labour?
Qn: What is the hori­zon against which CEA pro­grams are act­ing?

On research

The pri­mary less visi­ble ac­tivity of both GWWC and 80,000 hours is re­search. For GWWC, there are ques­tions to be re­solved about how best to Earn to Give, whether there are other ac­tivi­ties which are less im­me­di­ately fis­cal but of high im­pact, and broadly how to iden­tify near op­ti­mal op­por­tu­ni­ties for dona­tion. For 80,000 hours, there is a need to es­tab­lish how to op­ti­mise ca­reer paths for a broad set of po­ten­tial ter­mi­nal goals. Nei­ther pro­ject ap­pears to be bear­ing visi­ble fruit. In con­ver­sa­tion with Will Crouch, he ob­served that 80,000 hours don’t know much about the burnout rates of var­i­ous ca­reers, the wage pro­gres­sions or the like­li­hood of ca­reer pro­gres­sion.

At pre­sent, this means that 80,000 hours is not pub­li­cly pre­sent­ing things which are bet­ter than other sources of ad­vice. There is a need for the best cur­rent knowl­edge to be available quickly; there are peo­ple who are de­cid­ing ca­reers now who are un­likely to do re­li­ably bet­ter than av­er­age on the ba­sis of the in­for­ma­tion that 80,000 hours has made pub­lic. It seems im­plau­si­ble that new re­sults are re­li­ably com­ing in so quickly that the time spent pub­lish­ing the in­ter­nal state of the art will sub­stan­tively slow down fur­ther im­prove­ments. There is a strong sense in which they are be­ing graded on their speed, with pub­li­ca­tion be­ing the me­di­a­tor of im­pact. It also seems plau­si­ble that the pub­lish­ing and re­search are sub­stan­tially or­thog­o­nal, and would use differ­ent peo­ple. Hence from the out­side, the lack of pub­lished con­crete ad­vice seems to be a sub­stan­tial rea­son to think that there is no in­ter­nal art.

Qn: What is 80,000 hours pro­duc­ing with their cur­rent re­search time? What is the planned sched­ule? What con­sti­tutes suc­cess?

There is a similar con­cern with the out­put of GWWC. Of their listed pa­pers, only one (by Toby Ord) is sub­stan­tive. The re­main­der are not writ­ten as if there is a press­ing need to have re­sults that are con­cise, clear and bet­ter than other available ma­te­ri­als. For ex­am­ple, there is an ex­tended ar­ti­cle on in­vest­ing vs. giv­ing, and an­other on the dis­tinc­tion be­tween in­come and hap­piness. The former does lit­tle more than list fac­tors that might be rele­vant, with no at­tempt to dis­cern which of these effects are largest or a sense of what ranges of rea­son­able look­ing as­sump­tions give. The sec­ond ob­serves cor­rectly that the im­pact of mon­e­tary loss on donors may be over­es­ti­mated, but then doesn’t even ques­tion how im­pacts on re­cip­i­ents should be con­verted into he­do­nic terms. As a doc­u­ment, it seems to have been writ­ten to con­vince rather than elu­ci­date truth. Nei­ther pa­per drives an up­date to a be­lief that the cur­rent re­searchers at GWWC are effec­tively seek­ing to iden­tify close to op­ti­mal op­por­tu­ni­ties or to rea­son co­her­ently about the im­pact of in­ter­ven­tions.

More wor­ry­ing is the ab­solute lack of ma­te­rial. Whilst the num­ber of ac­tive re­searchers is difficult to dis­cern from the web­site, it seems plau­si­ble that GWWC has had at least 6 peo­ple re­search­ing for it for at least the last year. There is no match­ing level of out­put; in academe one would ex­pect to see sev­eral pa­pers per year per per­son, and the pri­mary claim of GWWC is that there are low hang­ing fruit in terms of the op­ti­mi­sa­tion of dona­tion and the abil­ity of peo­ple to donate. So a pri­ori, if GWWC was effi­ciently re­search­ing, one would ex­pect it to be find­ing and pub­li­cis­ing their re­sults.

Qn: What is GWWC pro­duc­ing with their cur­rent re­search time? What is the planned sched­ule? What con­sti­tutes suc­cess?

80,000 Hours

Im­pact of the 80,000 hours advice

In con­ver­sa­tion with Will, he as­serted that on the ba­sis of self-re­ports, some­thing like 20-25% of those in­volved in 80K have changed or sub­stan­tially rethought their ca­reer choice. This im­plies im­me­di­ately that 75-80% haven’t, and in prac­tise that num­ber will be higher care of the self-re­port­ing. This sub­stan­tially re­duces the likely im­pact of 80,000 hours as a pro­gram. In­deed, it seems to be a near fatal prob­lem for GWWC, in that if the 80,000 hours pop­u­la­tion is rep­re­sen­ta­tive of pledges, then most of the GWWC pledges are earn­ing in line with typ­i­cal post grads, which makes it much harder to raise the mean value of each pledge to £250K as is re­quired.

Meth­ods of achiev­ing this im­pact do not seem to be well at­tested. Will was asked what the in­ter­nal value of a paid worker in Cam­bridge might be. A broad re­sponse was that it might im­prove the abil­ity to give ad­vice, but it was not sug­gested that this was based on hard data. This is a lit­tle trou­bling, be­cause it im­plies that the effec­tive­ness of 1-1 Skype in­ter­ven­tions or 1-1 in per­son in­ter­ven­tions are not known on a per hour ba­sis. Ab­sent this kind of data, it’s difficult to see how 80,000 hours can be effec­tively op­ti­mis­ing their im­pact.

Qn: Does 80,000 hours have data on the rel­a­tive effec­tive­ness of their ac­tivi­ties?
Qn: How does CEA square a lack of re­ported ca­reer changes with GWWC’s num­bers, given back­ground over-life earn­ings?

Con­tent of 80,000 hours’ advice

In con­ver­sa­tion, earn­ing to give was sug­gested as be­ing the baseline to mea­sure against. Will noted ex­plic­itly that it’s hard to know what kinds of ca­reers are sub­stan­tively bet­ter than oth­ers in a data-driven way. He was then very quick to hedge that by say­ing that of course re­search was valuable, and of course poli­ti­cal ac­tivism could be valuable, and of course be­ing a pro­gram man­ager at the world bank could be valuable (which would nat­u­rally re­quire you do a PhD first), and of course be­ing an en­trepreneur could be valuable. It was not sug­gested that clearly at most one of these was op­ti­mal, or that peo­ple might ul­ti­mately be in a po­si­tion where they trade off what they would have cho­sen to do in iso­la­tion against world op­ti­mis­ing goals. We came away from this with the con­cern that 80,000 hours is not be­ing epistem­i­cally vi­cious, and so is not will­ing to say things that might cause peo­ple to be un­happy. In par­tic­u­lar, it seemed that there was more pres­sure to pre­serv­ing the fuzzies that peo­ple were get­ting out of be­ing af­fili­ated to 80,000 hours than there was to make the ad­vice good, and so most po­ten­tial ca­reer paths were deemed to be OK.

Qn: Does 80,000 hours offer in­for­ma­tion that causes a sub­stan­tial re­duc­tion in the space of ca­reers that are con­sid­ered op­ti­mal?
Qn: How does 80,000 hours square a lack of re­ported ca­reer changes with their ad­vice be­ing good?

The 80,000 hours pledge

It was noted that the pledge had been sub­stan­tially weak­ened, to “I in­tend, at least in part, to use my ca­reer in an effec­tive way to make the world a bet­ter place.”. My rec­ol­lec­tion says that it used to be more like “I will use my ca­reer to most effec­tively re­duce global poverty”. There wasn’t any par­tic­u­lar defence of the choice of word­ing or any in­di­ca­tion that there had been deep thought about pre­cisely what that pledge should con­sti­tute.

The core mechanism by which 80,000 hours or GWWC will achieve long term im­pact has to be main­tain­ing peo­ple’s de­sire to act over a long pe­riod. In turn, it seems that the pri­mary in­ter­me­di­ate goal is to build a strong so­cial struc­ture to en­courage ad­her­ence to these pledges. The pledges are then the key totems around which a com­mu­nity will be built, and so there should be mas­sive pres­sure to op­ti­mise these and the sur­round­ing so­cial struc­tures. This does not seem to have oc­curred.

Qn: What are the de­sign de­ci­sions be­hind the pledge, and what mo­ti­vated the change in pledge?
Qn: To what ex­tent is the word­ing of the pledge thought to be im­por­tant?

Scope and goals

Speed of growth

It was stated that around 13 of the Oxford un­der­grad­u­ate pop­u­la­tion (~4000 peo­ple) are on the mailing lists. Of that, there are around 300 mem­bers and a few dozen are com­ing to each event. By com­par­i­son, en­ter­pris­ing col­lege so­cieties in Cam­bridge (TMS, TCSS) have well in ex­cess of 1000 un­der­grad­u­ates on their mailing lists, and get 80-100 peo­ple to their talks. When TCSS ad­ver­tised an event to 13 of Cam­bridge, up­wards of 600 peo­ple at­tended. From some or­gani­sa­tional point of view, 80,000 hours Oxford could prob­a­bly ex­tract an­other fac­tor of 5-10 out of its talk at­ten­dance. Whilst that won’t fac­tor through di­rectly to the pledges, it seems un­likely that there would not be sub­stan­tial growth there. In both of the Cam­bridge so­cieties, the op­er­at­ing scale of the so­ciety has been dou­bled in a sin­gle year, by en­sur­ing a re­li­able stream of events and get­ting net­works in place to ad­ver­tise widely.

It does not seem like the or­gani­sa­tions are op­ti­mis­ing for growth and re­ten­tion of a pop­u­la­tion of at­ten­dees. This would provide a pool of peo­ple broadly on board with the aims of the or­gani­sa­tions, and sub­stan­tially en­riched for likely pledges. It is very plain that such op­ti­mi­sa­tion has not been cod­ified and sent to other new chap­ters; the Cam­bridge GWWC chap­ter does not be­have as if such guidance ex­ists.

Qn: What op­ti­mi­sa­tion has GWWC /​ 80,000 hours at­tempted in terms of the struc­ture of their chap­ters?

Ambition

Tak­ing a larger scale view, lots of these con­cerns ul­ti­mately cash out in a con­cern that a large frac­tion of the peo­ple in­volved with 80,000 hours or GWWC be­have like dilet­tantes. There is an ap­par­ent de­sire to feel com­fortable about ca­reer choice, think about deal­ing with poverty and get in­volved with 501(c)(3)’s/​NGO’s/​UK char­i­ties. How­ever as or­gani­sa­tions, they are not be­hav­ing as we would ex­pect for a bunch of peo­ple that se­ri­ously ex­pect to vec­tor hun­dreds of mil­lions of pounds over the next decade, which is what con­tinued lin­ear growth would im­ply.

Nor do they seem to act as if they wish to se­ri­ously op­ti­mise the world. For ex­am­ple, the world bank throws ~$43B/​year around. Which is eas­ier: To up­scale GWWC by a fac­tor of ~17000, or dou­ble the mean effec­tive­ness of the world bank? This should not be a hy­po­thet­i­cal ques­tion; it should be an­swered. There doesn’t seem to be an ac­cep­tance that large so­cial struc­tures are go­ing to be needed to sup­port GWWC style dona­tion for a life­time, in the fash­ion of say the ro­tary clubs.

Qn: Where does CEA see its pro­jects in 10 years? 20? 40?