[Link] Social interventions gone wrong

A piece I saw that Ben­jamin Todd adapted from THINK’s mod­ule on char­ity as­sess­ment. Some of you may re­call the net­work’s re­cent launch.

Lots of so­cial in­ter­ven­tions end up do­ing more harm than good. Many more make no differ­ence at all, and are just a waste of re­sources. At times, we’ve prob­a­bly ar­gued with friends about which in­ter­ven­tions we’d like to see, and which we wouldn’t. But are we any good at judg­ing what’s likely to work?

Here’s a cool bit of con­tent adapted from THINK. Try and guess which of these eight pro­grams made a differ­ence, which had no effect, and which made things worse.

ciper­goth said that it should be em­pha­sised that this isn’t a trick ques­tion where the an­swer is they all worked or none did.

Round #1: Scared Straight

Pro­gram de­scrip­tion: “In the 1970s, in­mates serv­ing life sen­tences at a New Jersey (USA) prison be­gan a pro­gram to ‘scare’ or de­ter at‐risk or delin­quent chil­dren from a fu­ture life of crime. The pro­gram, known as ‘Scared Straight’, fea­tured as its main com­po­nent an ag­gres­sive pre­sen­ta­tion by in­mates to ju­ve­niles vis­it­ing the prison fa­cil­ity. The pre­sen­ta­tion de­picted life in adult pris­ons, and of­ten in­cluded ex­ag­ger­ated sto­ries of rape and mur­der … The pro­gram re­ceived con­sid­er­able and fa­vor­able me­dia at­ten­tion and was soon repli­cated in over 30 ju­ris­dic­tions na­tion­wide … Although the harsh and some­times vul­gar pre­sen­ta­tion in the ear­lier New Jersey ver­sion is the most fa­mous, in­mate pre­sen­ta­tions are now some­times de­signed to be more ed­u­ca­tional than con­fronta­tional but with a similar crime pre­ven­tion goal. Some of these pro­grams fea­tured in­ter­ac­tive dis­cus­sions be­tween the in­mates and ju­ve­niles, also referred to as ‘rap ses­sions.’(2)

Did the pro­gram de­crease the rate of ju­ve­nile crime?

Round #2: Nurse‐Fam­ily Partnership

Pro­gram de­scrip­tion: “The Nurse‐Fam­ily Part­ner­ship pro­gram pro­vides nurse home vis­its to preg­nant women with no pre­vi­ous live births, most of whom are i) low‐in­come, ii) un­mar­ried, and iii) teenagers. The nurses visit the women ap­prox­i­mately once per month dur­ing their preg­nancy and the first two years of their chil­dren’s lives. The nurses teach i) pos­i­tive health re­lated be­hav­iors, ii) com­pe­tent care of chil­dren, and iii) ma­ter­nal per­sonal de­vel­op­ment (fam­ily plan­ning, ed­u­ca­tional achieve­ment, and par­ti­ci­pa­tion in the work­force). The pro­gram costs ap­prox­i­mately $12,500 per woman over the three years of vis­its (in 2010 dol­lars).”(6)

Did the pro­gram im­prove the qual­ity of child care?

Round #3: Drug Abuse Re­sis­tance Ed­u­ca­tion (DARE)

Pro­gram de­scrip­tion: “DARE is a highly‐struc­tured sub­stance‐abuse pre­ven­tion pro­gram taught by uniformed po­lice officers … The pro­gram is typ­i­cally pro­vided over the course of 10‐20 weekly hour‐long ses­sions, dur­ing which the po­lice officers use lec­tures, class dis­cus­sion, role plays, and home­work as­sign­ments to i) teach stu­dents about sub­stance use and its effects, ii) teach stu­dents de­ci­sion‐mak­ing and peer pres­sure re­sis­tance skills, and iii) boost stu­dents’ self‐es­teem. Prior to teach­ing, the po­lice officers take an 80‐hour train­ing course on teach­ing tech­niques, class­room man­age­ment, and the DARE cur­ricu­lum … DARE costs ap­prox­i­mately $130 (in 2004 dol­lars) per stu­dent and, as of 2001, was op­er­at­ing in 75% of Amer­i­can school dis­tricts.”(8)

Did the pro­gram de­crease the rate of drug use?

Rounds #4 and #5: 21st Cen­tury Com­mu­nity Learn­ing Centers

Pro­gram de­scrip­tion: “21st Cen­tury Com­mu­nity Learn­ing cen­ters is a large ($1 billion per year) US Depart­ment of Ed­u­ca­tion pro­gram which funds op­tional af­ter‐school pro­grams for el­e­men­tary and mid­dle school stu­dents in mostly high‐poverty schools. Key goals of the pro­gram are to i) provide stu­dents with a safe place af­ter school, and ii) im­prove their aca­demic perfor­mance. Re­cip­i­ents of pro­gram funds (ie, school dis­tricts and/​or non‐profit ed­u­ca­tional/​com­mu­nity or­ga­ni­za­tions) are re­quired to provide aca­dem­i­cally fo­cused “ex­tended learn­ing ac­tivi­ties” (e.g., in­struc­tional en­rich­ment pro­grams, tu­tor­ing, or home­work as­sis­tance). Most cen­ters also offer en­rich­ment/​recre­ational ac­tivi­ties such as mar­tial arts, sports, dance, art and/​or mu­sic … (Ele­men­tary school) cen­ters vary in the ac­tivi­ties they offer and other key fea­tures, and thus com­prise a range of af­ter‐school in­ter­ven­tions rather than a sin­gle in­ter­ven­tion. In a typ­i­cal cen­ter,

  1. stu­dents may spend an hour do­ing home­work and hav­ing a snack, an hour on ad­di­tional aca­demic ac­tivity (eg, a les­son or work­ing in a com­puter lab), and an hour do­ing recre­ational or cul­tural ac­tivi­ties;

  2. the cen­ter’s staff are a mix­ture of cer­tified teach­ers, in­struc­tional aides, and rep­re­sen­ta­tives of com­mu­nity youth or­ga­ni­za­tions;

  3. the cen­ter is open 4‐5 days per week for three hours af­ter school, and serves ap­prox­i­mately 85 stu­dents per day; and

  4. the av­er­age stu­dent at­tends the cen­ter 2‐3 days per week.

Cen­ters spend ap­prox­i­mately $1,000 (in 2005 dol­lars) on each en­rol­led stu­dent per year.”(10)

Did the pro­gram in­crease the stu­dents’ aca­demic achieve­ment?

Did the pro­gram im­prove the be­havi­oural prob­lems at the schools?

Round #6: Even Start Fam­ily Liter­acy program

Pro­gram de­scrip­tion: “The Even Start pro­gram is in­tended to ‘help break the cy­cle of poverty and illiter­acy by im­prov­ing the ed­u­ca­tional op­por­tu­ni­ties of the na­tion’s low‐in­come fam­i­lies by in­te­grat­ing early child­hood ed­u­ca­tion, adult liter­acy or adult ba­sic ed­u­ca­tion, and par­ent­ing ed­u­ca­tion into a unified fam­ily liter­acy pro­gram’. In 2000‐2001, there were 855 Even Start pro­jects serv­ing 31,896 fam­i­lies … Even Start grantees had con­sid­er­able flex­i­bil­ity in de­sign­ing ser­vices to meet the needs of the low‐in­come fam­i­lies, but all were re­quired to offer four ser­vices:

  1. adult ed­u­ca­tion to de­velop ba­sic ed­u­ca­tional and liter­acy skills;

  2. early child­hood ed­u­ca­tion ser­vices to provide de­vel­op­men­tally ap­pro­pri­ate ser­vices to help pre­pare chil­dren for school;

  3. par­ent­ing ed­u­ca­tion to help par­ents sup­port the ed­u­ca­tional growth of their chil­dren; and

  4. par­ent‐child liter­acy ac­tivi­ties.”(13)

Did the pro­gram in­crease liter­acy?

Round #7: Big Brothers Big Sisters

Pro­gram de­scrip­tion: “Big Brothers Big Sisters’ com­mu­nity‐based men­tor­ing pro­gram matches youths aged 6‐18, pre­dom­i­nantly from low‐in­come, sin­gle‐par­ent house­holds, with adult vol­un­teer men­tors who are typ­i­cally young (20‐34) and well‐ed­u­cated (the ma­jor­ity are col­lege grad­u­ates) … The men­tor and youth typ­i­cally meet for 2‐4 times per month for at least a year, and en­gage in ac­tivi­ties of their choos­ing (e.g., study­ing, cook­ing, play­ing sports). The typ­i­cal meet­ing lasts 3‐4 hours … For the first year, Big Brothers Big Sisters case work­ers main­tain monthly con­tact with the men­tor, as well as the youth and his or her par­ent, to in­sure a pos­i­tive men­tor‐youth match, and to help re­solve any prob­lems in the re­la­tion­ship. Men­tors are en­couraged to form a sup­port­ive friend­ship with the youths, as op­posed to mod­ify­ing the youth’s be­hav­ior or char­ac­ter… In 2008, Big Brothers Big Sisters served 255,000 youths and 470 agen­cies na­tion­wide. The na­tional av­er­age cost of mak­ing and sup­port­ing a match is ap­prox­i­mately $1,300 in 2009 dol­lars.”(14)

Did the pro­gram de­crease drug use and vi­o­lent be­hav­ior?

Round #8: Top 16 Ed­u­ca­tional Software

Pro­gram de­scrip­tion: “In the No Child Left Be­hind Act of 2002, Congress called for a rigor­ous study of the effec­tive­ness of ed­u­ca­tional tech­nol­ogy for im­prov­ing stu­dent aca­demic achieve­ment … In fall 2003, de­vel­op­ers and ven­dors of ed­u­ca­tional tech­nol­ogy prod­ucts re­sponded to a pub­lic in­vi­ta­tion and sub­mit­ted prod­ucts for pos­si­ble in­clu­sion in the na­tional study. Math­e­mat­ica Policy Re­search, Inc. staff se­lected 40 of the 160 sub­mis­sions for fur­ther re­view by two pan­els of out­side ex­perts, one for read­ing prod­ucts and one for math prod­ucts … In Jan­uary 2004, (the US Depart­ment of Ed­u­ca­tion] con­sid­ered the panel’s recom­men­da­tions and se­lected 16 prod­ucts for the study. In se­lect­ing prod­ucts, (the US Depart­ment of Ed­u­ca­tion) grouped them into four ar­eas:

  1. early read­ing (first grade),

  2. read­ing com­pre­hen­sion (fourth grade),

  3. pre‐alge­bra (sixth grade), and

  4. alge­bra (ninth grade).

The prod­ucts ranged widely in their in­struc­tional ap­proaches and how long they had been in use. In gen­eral, how­ever, the crite­ria weighted the se­lec­tion to­wards prod­ucts that had ev­i­dence of effec­tive­ness from pre­vi­ous re­search, or, for newer prod­ucts, ev­i­dence that their de­signs were based on ap­proaches found to be effec­tive by re­search. Twelve of the six­teen prod­ucts had re­ceived awards or been nom­i­nated for awards (some as re­cently as 2006) by trade as­so­ci­a­tions, me­dia, teach­ers, or par­ents.”(15)

Did the pro­gram im­prove test scores?

Here are the an­swers!

Round #1: Scared Straight

Nega­tive! Sev­eral ran­dom­ized con­trol­led tri­als have shown that Scared Straight had a nega­tive effect. Go­ing through Scared Straight made chil­dren more likely to com­mit crimes in the fu­ture (3). Fun fact: Scared Straight pro­grams are still be­ing run to­day (4), and peo­ple pro­mote them as be­ing effec­tive, de­spite the fact that they are harm­ful (5).

Round #2: Nurse‐Fam­ily Partnership

Pos­i­tive! Three ran­dom­ized con­trol­led tri­als have shown that the Nurse‐Fam­ily Part­ner­ship had a pos­i­tive effect. The pro­gram led to a re­duc­tion in child abuse/​ne­glect, child in­juries (20‐50% re­duc­tion) and an im­prove­ment in cog­ni­tive/​ed­u­ca­tional out­comes for chil­dren of moth­ers with low men­tal health/​con­fi­dence/​in­tel­li­gence (e.g., 6 per­centile point in­crease in grade 1‐6 in read­ing/​math achieve­ment) (7).

Round #3: Drug Abuse Re­sis­tance Ed­u­ca­tion (DARE)

No effect!Two ran­dom­ized con­trol­led tri­als have shown that DARE did not have an effect on the rate of drug use among par­ti­ci­pants. The rate of drug use did not in­crease or de­crease (9).

Round #4: 21st Cen­tury Com­mu­nity Learn­ing Centers

No effect! A ran­dom­ized con­trol­led trial has shown that the 21st Cen­tury Com­mu­nity Learn­ing Cen­ters had no effect on par­ti­ci­pat­ing stu­dents’ aca­demic perfor­mance. Stu­dents who par­ti­ci­pated were nei­ther helped nor harmed by the pro­gram.(11)

Round #5: 21st Cen­tury Com­mu­nity Learn­ing Centers

Nega­tive! A ran­dom­ized con­trol­led trial has shown that the 21st Cen­tury Com­mu­nity Learn­ing Cen­ters caused an in­crease in the be­hav­ioral prob­lems of par­ti­ci­pat­ing stu­dents (12).

Round #6: Even Start Fam­ily Liter­acy Program

No effect! A ran­dom­ized con­trol­led trial on a sub­set of Even Start pro­grams found no ev­i­dence of an in­crease or de­crease in liter­acy in par­ents or chil­dren (17).

Round #7: Big Brothers Big Sisters

Pos­i­tive! A ran­dom­ized con­trol­led trial has shown that Big Brothers Big Sisters caused youths to be 46% less likely to have started us­ing ille­gal drugs, 27% less likely to have started us­ing al­co­hol, 32% less likely to have hit some­one in the pre­vi­ous year and fewer days of skip­ping school dur­ing the past year (18).

Round #8: Top 16 Ed­u­ca­tional Software

No effect!The study de­scribed was a ran­dom­ized con­trol­led trial, and showed that the soft­ware did not make a no­tice­able differ­ence in any of the cat­e­gories. It did not help or hurt with 1) early read­ing (first grade), 2) read­ing com­pre­hen­sion (fourth grade), 3) pre‐ alge­bra (sixth grade), or 4) alge­bra (ninth grade) (19).

How did you do?

If you got 7-8 right, there’s less than a 1% chance you were guess­ing. If you got 5-6 right, there was only an 8.5% chance you were guess­ing, so it might be skill. If you got 1-4 right, then you did no bet­ter than ran­domly guess­ing. If you got zero right … we could get use­ful in­for­ma­tion by always do­ing the op­po­site of what you do.

The effects of so­cial in­ter­ven­tions are ex­tremely com­plex. All of these pro­grams sound good, but un­in­tended con­se­quences can get in the way. It’s very difficult to work out what’s go­ing to be suc­cess­ful ahead of time. In­stead, we need to test, mea­sure the re­sults, and take it from there.

I thought Round 2 would have no effect and ex­pected Round #5 to have no effect not a nega­tive one, I got 6 out of 8 cor­rect. How well did you do?

I recom­mend check­ing out the links and refer­ences. Gw­ern’s com­ment there was also in­ter­est­ing.