Q for GiveWell: What is GiveDirectly’s mechanism of action?

I first wrote up the fol­low­ing post, then hap­pened to run into Holden Karnofsky in per­son and asked him a much-short­ened form of the ques­tion ver­bally. My at­tempt to re­count Holden’s ver­bal re­ply is also given fur­ther be­low. I was mod­er­ately im­pressed by Holden’s re­sponse be­cause I had not thought of it when list­ing out pos­si­ble replies, but I don’t un­der­stand yet why Holden’s re­sponse should be true. Since GiveWell has re­cently posted about ob­jec­tions to GiveDirectly and replies, I de­cided to go ahead and post this now.

A ques­tion for GiveWell:

Your cur­rent #2 top-rated char­ity is GiveDirectly, which gives one-time gifts of $1000 over 9 months, di­rectly to poor re­cip­i­ents in Kenya via M-PESA.

Givewell tries for high stan­dards of ev­i­dence of effi­cacy and cost-effec­tive­ness. As I un­der­stand it, you don’t just want the char­ity to be ar­guably cost effec­tive, you want a very high prob­a­bil­ity that the char­ity is cost-effec­tive.

The main ev­i­dence I’ve seen cited for di­rect giv­ing is that the re­cip­i­ents who re­ceived the $1000 are then sub­stan­tially bet­ter off 9 months later com­pared to peo­ple who aren’t.

While I can imag­ine ar­gu­ments that could re­pair the ob­vi­ous ob­jec­tion to this rea­son­ing, I haven’t seen yet how the re­sult­ing ev­i­dence about cost-effec­tive­ness could rise again to the epistemic stan­dards one would ex­pect of Givewell’s #2 ev­i­dence-based char­ity.

The ob­vi­ous ob­jec­tion is as fol­lows: Sup­pose the Kenyan gov­ern­ment sim­ply printed new shillings and handed out $1000 of such shillings to the same re­cip­i­ents tar­geted by GiveDirectly. Although the re­cip­i­ents would be bet­ter off than non-re­cip­i­ents, this might not re­flect any im­prove­ment in net util­ity in Kenya be­cause no new re­sources were cre­ated by print­ing the money.

There are of course ob­vi­ous replies to this ob­vi­ous ob­jec­tion:

(1) Be­cause the shillings handed out by GiveDirectly are pur­chased on the for­eign cur­rency ex­change mar­ket us­ing U. S. dol­lars, and would oth­er­wise have been spent in Kenya in other ways, we should not ex­pect any in­fla­tion of the shilling, and should ex­pect an in­crease in Kenyan con­sump­tion of for­eign goods cor­re­spond­ing to the in­creased price of shillings im­plied by GiveDirectly adding their marginal de­mand to the auc­tion and thereby rais­ing the marginal price of all shillings sold. The pri­mary mechanism of ac­tion by which GiveDirectly benefits Kenya is by rais­ing the price of shillings in the for­eign ex­change mar­ket and mak­ing more hard cur­rency available to sel­l­ers of shillings. So far as I can tell, this ar­gu­ment ought to gen­er­al­ize: Any ar­gu­ment that the Kenyan gov­ern­ment could not ac­com­plish most of the same good by print­ing shillings will mean that the pri­mary mechanism of GiveWell’s effec­tive­ness must be the U.S. dol­lars be­ing ex­changed for the shillings on the for­eign cur­rency mar­ket. This in turn means that GiveDirectly could ac­com­plish most of its good by buy­ing the same shillings on the for­eign cur­rency mar­ket and burn­ing them.

(Or to sharpen the to­tal point of this ar­ti­cle: The sum of the good ac­com­plished by GiveDirectly should equal:

  • The good ac­com­plished by the Kenyan gov­ern­ment print­ing shillings and dis­tribut­ing them to the same re­cip­i­ents;

  • plus the good ac­com­plished by GiveDirectly then pur­chas­ing shillings on the for­eign ex­change mar­ket us­ing US dol­lars, and burn­ing them.

In­deed, since these mechanisms of ac­tion seem mostly in­de­pen­dent, we ought to be able to state a per­centage of good ac­com­plished which is allegedly at­tributed to each, sum­ming to 1. E.g. maybe 80% of the good would be achieved by print­ing shillings and dis­tribut­ing them to the same re­cip­i­ents, and 20% would be achieved by pur­chas­ing shillings on the for­eign ex­change mar­ket and burn­ing them. But then we have mostly the same ques­tions as be­fore about how to gen­er­ate wealth by print­ing shillings.)

(2) Inequal­ity in Kenya is such that re­dis­tribut­ing the sup­ply of shillings to­ward the very poor in­creases util­ity in Kenya. Thus the Kenyan gov­ern­ment could ac­com­plish as much good as GiveDirectly by print­ing an equiv­a­lent num­ber of shillings and giv­ing them to the same re­cip­i­ents. This would cre­ate in­fla­tion that is a loss to other Kenyans, some of them also very poor, but so much of the shilling sup­ply is held by the rich that the net re­sults are fa­vor­able. Print­ing shillings can cre­ate hap­piness be­cause it shifts re­sources from mak­ing speed­boats for the rich to mak­ing cor­ru­gated iron roofs for the poor.

(It would be nice if the Kenyan gov­ern­ment just printed shillings for GiveDirectly to use, but this the Kenyan gov­ern­ment will not re­al­is­ti­cally do. Effec­tive al­tru­ists must live in the real world, and in the real world GiveDirectly will only ac­com­plish its goals with the aid of effec­tive al­tru­ists. One can­not live in the should-uni­verse where Kenya’s gov­ern­ment is tak­ing up the bur­den. Effec­tive al­tru­ists should rea­son as if the Kenya gov­ern­ment con­sists of plas­tic dolls who can­not be the lo­cus of re­spon­si­bil­ity in­stead of them—that’s heroic episte­mol­ogy 101. Maybe there will even­tu­ally be re­turns on lob­by­ing for Min­i­mum Guaran­teed In­come in Kenya if the pro­grams work, but that’s for to­mor­row, not right now.)

(3) Like the Euro­pean Union, Kenya is not print­ing enough shillings un­der stan­dard eco­nomic the­ory. (I have no idea if this is plau­si­bly true for Kenya in par­tic­u­lar.) If the gov­ern­ment printed shillings and gave them to the same re­cip­i­ents, this would cre­ate real wealth in Kenya be­cause the econ­omy was op­er­at­ing be­low ca­pac­ity and ve­loc­ity of trade would pick up. The shillings pur­chased by GiveDirectly would oth­er­wise have stayed in bank ac­counts rather than go­ing to other Kenyans. Note that this con­tra­dicts the ar­gu­ment step in (1) where we said that the pur­chased shillings would oth­er­wise have been spent el­se­where, so you should have ques­tioned one ar­gu­ment step or the other.

(4) Village moneylen­ders and bosses can suc­cess­fully ex­tract most sur­plus gen­er­ated within their villages by rais­ing rents or de­mand­ing bribes. The only way that in­di­vi­d­u­als can es­cape the grasp of moneylen­ders and ren­tiers is with a one-time gift that was not ex­pected and which the moneylen­ders and bosses could not ar­range to cap­ture. The gov­ern­ment could ac­com­plish as much good as GiveDirectly by print­ing the same num­ber of shillings and giv­ing them to the same peo­ple in an un­pre­dictable pat­tern. This would cre­ate some in­fla­tion but village moneylen­ders or bosses would ease off on peo­ple from whom they couldn’t ex­tract as much value, whereas the one-time gift re­cip­i­ents can pur­chase cap­i­tal goods that will make them per­ma­nently bet­ter off in ways that don’t al­low the new value to be ex­tracted by moneylen­ders or bosses.

If I re­call cor­rectly, GiveDirectly uses the ex­am­ple of a fam­ily us­ing some of the gift money to pur­chase a cor­ru­gated iron roof. From my per­spec­tive the ob­vi­ous ob­jec­tion is that they could just be pur­chas­ing a cor­ru­gated iron roof that would’ve gone to some­one else and rais­ing the prices of roofs. (1) says that Kenya has more for­eign ex­change on hands and can im­port, not one more cor­ru­gated iron roof, but a va­ri­ety of other for­eign goods; (2) says that the re­sources used in the cor­ru­gated iron roof would oth­er­wise have been used to make a speed­boat; (3) says that a new trade takes place in which some­body makes a cor­ru­gated iron roof that wouldn’t have been man­u­fac­tured oth­er­wise; and (4) says that the village moneylen­ders usu­ally ad­just their in­ter­est rates so as to pre­vent any­one from sav­ing up enough money to buy a cor­ru­gated iron roof.

The trou­ble is that all of these mechanisms of ac­tion seem much harder to mea­sure and be sure of, than the mea­surable out­comes for gift re­cip­i­ents vs. non-re­cip­i­ents.

To re­it­er­ate, the sum of the good ac­com­plished by GiveDirectly should equal the good ac­com­plished by the Kenyan gov­ern­ment print­ing shillings and dis­tribut­ing them to the same re­cip­i­ents, plus the good ac­com­plished by GiveDirectly pur­chas­ing shillings on the for­eign ex­change mar­ket us­ing US dol­lars and then burn­ing them. It seems to me to be difficult to ar­rive at a state of strong ev­i­dence about ei­ther of the two terms in this sum, with re­spect to any mechanism of ac­tion I’ve thought of so far.

With re­spect to the sec­ond term in this sum: GiveDirectly buy­ing shillings on the for­eign ex­change mar­ket and burn­ing them might cre­ate wealth, but it’s hard to see how you would mea­sure this over the rele­vant amounts, and no such ev­i­dence was cited in the recom­men­da­tion of GiveDirectly as the #2 char­ity.

With re­spect to the first term in this sum: Un­der the Bayesian defi­ni­tion of ev­i­dence, strong ev­i­dence is ev­i­dence we are un­likely to see when the the­ory is false. Even in the ab­sence of any mechanism whereby print­ing nom­i­nal shillings cre­ates hap­piness or wealth, we would still ex­pect to find that the wealth and hap­piness of gift re­cip­i­ents ex­ceeded the wealth of non-re­cip­i­ents. So mea­sur­ing that the gift re­cip­i­ents are wealthier and hap­pier is not strong or even medium ev­i­dence that print­ing nom­i­nal shillings cre­ates wealth, un­less I’m miss­ing some­thing here. Our pos­te­rior that print­ing shillings and giv­ing them to cer­tain peo­ple would cre­ate net wealth in any given quan­tity, should roughly equal our prior, af­ter up­dat­ing on the stated ex­per­i­men­tal ev­i­dence.

When I posed a short­ened form of this ques­tion to Holden Karnofsky, he replied (roughly, I am try­ing to rephrase from mem­ory):

It seems to me that this is a per­verse de­com­po­si­tion of the benefit ac­com­plished. There’s no in­fla­tion in the shilling be­cause you’re buy­ing them, and since this is true, de­com­pos­ing the benefit into an op­er­a­tion that does in­fla­tion­ary dam­age as a side effect, and then an­other op­er­a­tion that makes up for the in­fla­tion, is per­verse. It’s like crit­i­ciz­ing the Against Malaria Foun­da­tion based on a hy­po­thet­i­cal which in­volves the mosquito nets be­ing made from the flesh of ba­bies and then adding an­other effect which saves the lives of other ba­bies. Since this is a per­verse sum in­volv­ing a strange ex­tra side effect, it’s okay that we can’t get good es­ti­mates in­volv­ing ei­ther of the terms in it.

Please keep in mind that this is Holden’s off-the-cuff, non-writ­ten in-per­son re­sponse as rephrased by Eliezer Yud­kowsky from im­perfect mem­ory.

With that said, I’ve thought about (what I think was) Holden’s an­swer and I feel like I’m still miss­ing some­thing. I agree that if U.S. dol­lars were be­ing sent di­rectly to Kenyan re­cip­i­ents and used only to pur­chase for­eign goods, so that for­eign goods were be­ing di­rectly sent from the U.S. to Kenyan re­cip­i­ents, then im­prove­ment in mea­sured out­come for re­cip­i­ents com­pared to non-re­cip­i­ents would be an ap­pro­pri­ate met­ric, and that the de­com­po­si­tion would be per­verse. But if the re­ceived money, in the form of Kenyan shillings, is be­ing used pri­mar­ily to pur­chase Kenyan goods, and caus­ing those goods to be shipped to one villager rather than an­other while also pos­si­bly in­creas­ing ve­loc­ity of trade, rem­e­dy­ing in­equal­ity, and en­abling com­pletely differ­ent ac­tors to buy some amount of for­eign goods, then I hon­estly don’t un­der­stand why this sce­nario should have the same causal mechanisms as the sce­nario where for­eign goods are be­ing shipped in from out­side the coun­try. And then I hon­estly don’t un­der­stand why mea­sured im­prove­ments for one Kenyan over an­other should be a good proxy for ag­gre­gate welfare change to the coun­try.

I may be miss­ing some­thing that an economist would find ob­vi­ous or I may have mi­s­un­der­stood Holden’s re­ply. But to me, my sum seems like an ob­vi­ous causal de­com­po­si­tion of the effects in Kenya, nei­ther of whose terms can be es­ti­mated well. I don’t un­der­stand why I should ex­pect the un­cer­tainty in these two es­ti­mates to can­cel out when they are added; I don’t un­der­stand what back­ground causal model yields this con­clu­sion.

    To be clear, I per­son­ally would guess that the U.S. would be net bet­ter off, if the Fed­eral Re­serve di­rectly sent ev­ery­one in the U.S. with in­come un­der $20K/​year a one-time $6,000 check with the money phas­ing out at a 10% rate up to $80K/​year. This is be­cause, in or­der of im­por­tance:

    • I buy the analo­gous mar­ket mon­e­tarist ar­gu­ment (3) that the U.S. is print­ing too lit­tle money.

    • I buy the analo­gous ar­gu­ment (2) about in­equal­ity.

    • (How­ever, I also some­what sus­pect that some analo­gous form of (4) is go­ing on with poor peo­ple some­how sys­tem­at­i­cally hav­ing all but a cer­tain amount of value ex­tracted from them, which is in gen­eral how a mod­ern coun­try can have only 2% in­stead of 95% of the pop­u­la­tion be­ing farm­ers, and yet there are still peo­ple liv­ing hand-to-mouth. I would worry that a pre­dictable, uni­ver­sal one-time gift of $6K would not defeat this phe­nomenon, and that the gift money will just be ex­tracted again some­how. In the case of Min­i­mum Guaran­teed In­come, I would worry that the la­bor share of in­come will drop pro­por­tion­ally to small amounts of MGI as wages are just bid down by peo­ple who can live on less. Or some­thing. This would be a much longer dis­cus­sion and the ideas are much less sim­ple than the above two no­tions, prob­a­bly also less im­por­tant. I’m just men­tion­ing it again be­cause of my long-term puz­zle­ment with the ques­tion “Why are there still poor peo­ple af­ter agri­cul­tural pro­duc­tivity rose by a fac­tor of 100?”)

    What I wouldn’t say is that my be­lief in the above is as strong as my be­lief in, say, the in­tel­li­gence ex­plo­sion. I’d guess that the print­ing op­er­a­tion would do more good than harm, but it’s not what I would call a strong ev­i­dence-based con­clu­sion. If we’re go­ing to be okay with that stan­dard of ar­gu­ment gen­er­ally, then the top char­ity un­der that stan­dard of rea­son­ing, gen­er­ally and even­hand­edly ap­plied, ought to work out to some char­ity that does sci­ence and tech­nol­ogy re­search. (X-risk min­i­miza­tion might seem sub­stan­tially ‘weirder’ than that, but the best sci­ence-fund­ing char­i­ties should be only equally weird.) And I wouldn’t mea­sure the ex­cess of hap­piness of gift-re­cip­i­ents com­pared to non-re­cip­i­ents in a pi­lot pro­gram, and call this a good es­ti­mate of the net good if a Min­i­mum Guaran­teed In­come were uni­ver­sally adopted.

    So to re­it­er­ate, my ques­tion to Givewell is not “Why do you think GiveDirectly might maybe end up do­ing some good any­way?” but “Does GiveDirectly rise to the stan­dards re­quired for your #2 ev­i­dence-based char­ity?”