Some cruxes on impactful alternatives to AI policy work

Ben Pace and I (Richard Ngo) re­cently did a pub­lic dou­ble crux at the Berkeley REACH on how valuable it is for peo­ple to go into AI policy and strat­egy work: I was op­ti­mistic and Ben was pes­simistic. Dur­ing the ac­tual event, we didn’t come any­where near to find­ing a dou­ble crux on that is­sue. But af­ter a lot of sub­se­quent dis­cus­sion, we’ve come up with some more gen­eral cruxes about where im­pact comes from.

I found Ben’s model of how to have im­pact very in­ter­est­ing, and so in this post I’ve tried to ex­plain it, along with my dis­agree­ments. Ben liked the goal of writ­ing up a rough sum­mary of our po­si­tions and hav­ing fur­ther dis­cus­sion in the com­ments, so while he ed­ited it some­what he doesn’t at all think that it’s a perfect ar­gu­ment, and it’s not what he’d write if he spent 10 hours on it. He en­dorsed the word­ing of the cruxes as broadly ac­cu­rate.

(Dur­ing the dou­ble crux, we also dis­cussed how the heavy-tailed wor­ld­view ap­plies to com­mu­nity build­ing, but de­cided on this post to fo­cus on the ob­ject level of what im­pact looks like.)

Note from Ben: “I am not an ex­pert in policy, and have not put more than about 20-30 hours of thought into it to­tal as a ca­reer path. But, as I re­cently heard Robin Han­son say, there’s a com­mon situ­a­tion that looks like this: some peo­ple have a shiny idea that they think about a great deal and work through the de­tails of, that folks in other ar­eas are skep­ti­cal of given their par­tic­u­lar mod­els of how the world works. Even though the skep­tics have less de­tail, it can be use­ful to pub­li­cly say pre­cisely why they’re skep­ti­cal.

In this case I’m of­ten skep­ti­cal when folks tell me they’re work­ing to re­duce x-risk by fo­cus­ing on policy. Folks do­ing policy work in AI might be right, and I might be wrong, but it seemed like a good use of time to start a dis­cus­sion with Richard about how I was think­ing about it and what would change my mind. If the fol­low­ing dis­cus­sion causes me to change my mind on this ques­tion, I’ll be re­ally su­per happy with it.”

Ben’s model: Life in a heavy-tailed world

A heavy-tailed dis­tri­bu­tion is one where the prob­a­bil­ity of ex­treme out­comes doesn’t drop very rapidly, mean­ing that out­liers there­fore dom­i­nate the ex­pec­ta­tion of the dis­tri­bu­tion. Owen Cot­ton-Bar­ratt has writ­ten a brief ex­pla­na­tion of the idea here. Ex­am­ples of heavy-tailed dis­tri­bu­tions in­clude the Pareto dis­tri­bu­tion and the log-nor­mal dis­tri­bu­tion; other phrases peo­ple use to point at this con­cept in­clude ‘power laws’ (see Zero to One) and ‘black swans’ (see the re­cent SSC book re­view). Wealth is a heavy-tailed dis­tri­bu­tion, be­cause many peo­ple are clus­tered rel­a­tively near the me­dian, but the wealthiest peo­ple are mil­lions of times fur­ther away. Hu­man height and weight and run­ning speed are not heavy-tailed; there is no man as tall as 100 peo­ple.

There are three key claims that make up Ben’s view.

The first claim is that, since the in­dus­trial rev­olu­tion, we live in a world where the im­pact that small groups can have is much more heavy-tailed than in the past.

  • Peo­ple can af­fect in­cred­ibly large num­bers of other peo­ple wor­ld­wide. The In­ter­net is an ex­am­ple of a rev­olu­tion­ary de­vel­op­ment which al­lows this to hap­pen very quickly.

  • Star­tups are be­com­ing uni­corns un­prece­dent­edly quickly, and their val­u­a­tions are very heav­ily skewed.

  • The im­pact of global health in­ter­ven­tions is heavy-tail dis­tributed. So is fund­ing raised by Effec­tive Altru­ism—two donors have con­tributed more money than ev­ery­one else com­bined.

  • Google and Wikipe­dia qual­i­ta­tively changed how peo­ple ac­cess knowl­edge; peo­ple don’t need to ar­gue about ver­ifi­able facts any more.

  • Face­book qual­i­ta­tively changed how peo­ple in­ter­act with each other (e.g. FB events is a cru­cial tool for most lo­cal EA groups), and can swing elec­tions.

  • It’s not just that we got more ex­treme ver­sions of the same things, but rather that we can get un­fore­seen types of out­comes.

  • The books HPMOR and Su­per­in­tel­li­gence both led to mass changes in plans to­wards more effec­tive ends via the efforts of in­di­vi­d­u­als and small groups.

The sec­ond claim is that you should put sig­nifi­cant effort into re-ori­ent­ing your­self to use high-var­i­ance strate­gies.

  • Ben thinks that recom­mend­ing strate­gies which are safe and low-risk is in­sane when pul­ling out of a heavy-tailed dis­tri­bu­tion. You want ev­ery­one to be tak­ing high-var­i­ance strate­gies.

    • This is only true if the tails are long to the right and not to the left, which seems true to Ben. Most pro­jects tend to end up not pul­ling any use­ful lev­ers what­ever and just do noth­ing, but a few pull cru­cial lev­ers and solve open prob­lems or in­crease ca­pac­ity for co­or­di­na­tion.

  • Your in­tu­itions were built for the an­ces­tral en­vi­ron­ment where you didn’t need to be able to think about co­or­di­nat­ing hu­mans on the scale of mil­lions or billions, and yet you still rely heav­ily on the in­tu­itions you’re built with in nav­i­gat­ing the mod­ern en­vi­ron­ment.

  • Scope in­sen­si­tivity, fram­ing effects, taboo trade­offs, and risk aver­sion, are the key things here. You need to learn to train your S1 to un­der­stand math.

    • By de­fault, you’re not go­ing to spend enough effort find­ing or ex­e­cut­ing high-var­i­ance strate­gies.

  • We’re still only 20 years into the in­ter­net era. Things keep chang­ing qual­i­ta­tively, but Ben feels like ev­ery­one keeps ad­just­ing to the new tech­nol­ogy as if it were always this way.

  • Ben: “My straw model of the vast ma­jor­ity of peo­ple’s at­ti­tudes is: I guess Face­book and Twit­ter are just things now. I won’t spend time think­ing about whether I could build a plat­form as suc­cess­ful as those two but op­ti­mised bet­ter for e.g. in­tel­lec­tual progress or so­cial co­or­di­na­tion—ba­si­cally not just money.”

  • Ben: “I do note that never in his­tory has change been hap­pen­ing so quickly, so it makes sense that peo­ple’s in­tu­itions are off.”

  • While many in­sti­tu­tions have been re­designed to fit the in­ter­net, Ben feels like al­most no­body is try­ing to im­prove in­sti­tu­tions like sci­ence on a large scale, and that this is clear low-hang­ing al­tru­is­tic fruit.

  • The Open Philan­thropy Pro­ject has gone through this pro­cess of up­dat­ing away from safe, low-risk bets with GiveWell, to­ward hits-based giv­ing, which is an ex­am­ple of this kind of move.

The third claim is that AI policy is not a good place to get big wins nor to learn the rele­vant mind­set.

  • Ben: “On a first glance, gov­ern­ments, poli­tics and policy looks like the sort of place where I would not ex­pect to find highly ex­ploitable strate­gies, nor a place that will teach me the sorts of think­ing that will help me find them in fu­ture.”

  • Peo­ple in policy spend a lot of time think­ing about how to in­fluence gov­ern­ments. But gov­ern­ments are gen­er­ally too con­ven­tional and slow to reap the benefits of weird ac­tions with ex­treme out­comes.

  • Work­ing in policy doesn’t cul­ti­vate the right type of think­ing. You’re usu­ally in a con­ven­tional gov­ern­men­tal (or aca­demic) en­vi­ron­ment, stuck in­side the sys­tem, get­ting se­duced by lo­cal in­cen­tive gra­di­ents and pres­tige hi­er­ar­chies. You of­ten need to spend a long time work­ing your way to po­si­tions of ac­tual im­por­tance in the gov­ern­ment, which leaves you prone to value drift or over-spe­cial­i­sa­tion in the wrong thing.

    • At the very least, you have to op­er­ate on the lo­cal in­cen­tives as well as some­one who ac­tu­ally cares about them, which can be dam­ag­ing to one’s abil­ity to think clearly.

  • Poli­ti­cal land­scapes are not the sort of en­vi­ron­ment where peo­ple can eas­ily ig­nore the lo­cal so­cial in­cen­tives to fo­cus on long-term, global goals. Short term think­ing (elec­tion cy­cles, me­dia cov­er­age, etc) is not the sort of think­ing that lets you build a new in­sti­tu­tion over 10 years or more.

    • Ben: “When I’ve talked to se­nior poli­ti­cal peo­ple, I’ve of­ten heard things of the sort ‘We were work­ing on a big strat­egy to im­prove in­fras­truc­ture /​ in­ter­na­tional aid /​ tech policy etc, but then sud­denly pub­lic ap­proval changed and then we couldn’t make head­way /​ our party wasn’t in power /​ etc.’ which makes me think long term plan­ning is strongly dis­in­cen­tivised.”

  • One les­son of a heavy-tailed world is that sig­nals that you’re tak­ing safe bets are anti-sig­nals of value. Many peo­ple fol­low­ing a stan­dard aca­demic track say­ing “Yeah, I’m gonna get a mas­ters in pub­lic policy” sounds fine, sen­si­ble, and safe, and there­fore can­not be an ac­tive sign that you will do some­thing a mil­lion times more im­pact­ful than the me­dian.

The above is not a full, gears-level anal­y­sis of how to find and ex­ploit a heavy tail, be­cause al­most all of the work here lies in iden­ti­fy­ing the par­tic­u­lar strat­egy. Nev­er­the­less, be­cause of the con­sid­er­a­tions above, Ben thinks that tal­ented, agenty and ra­tio­nal peo­ple should be able in many cases to iden­tify places to win, and then ex­e­cute those plans, and that this is much less the case in policy.

Richard’s model: Busi­ness (mostly) as usual

I dis­agree with Ben on all three points above, to vary­ing de­grees.

On the first point, I agree that the dis­tri­bu­tion of suc­cess has be­come much more heavy-tailed since the in­dus­trial rev­olu­tion. How­ever, I think the dis­tri­bu­tion of suc­cess is of­ten very differ­ent from the dis­tri­bu­tion of im­pact, be­cause of re­place­ment effects. If Face­book hadn’t be­come the lead­ing so­cial net­work, then MyS­pace would have. If not Google, then Ya­hoo. If not New­ton, then Leib­niz (and if New­ton, then Leib­niz any­way). Prob­a­bly the al­ter­na­tives would have been some­what worse, but not sig­nifi­cantly so (and if they were, differ­ent com­peti­tors would have come along). The dis­t­in­guish­ing trait of moder­nity is that even a small differ­ence in qual­ity can lead to a huge differ­ence in earn­ings, via net­work effects and global mar­kets. But that isn’t par­tic­u­larly in­ter­est­ing from an x-risk per­spec­tive, be­cause money isn’t any­where near be­ing our main bot­tle­neck.

You might think that since Face­book has billions of users, their ex­ec­u­tives are a small group with a huge amount of power, but I claim that they’re much more con­strained by com­pet­i­tive pres­sures than they seem. Their suc­cess de­pends on the loy­alty of their users, but the big­ger they are, the eas­ier it is for them to seem un­trust­wor­thy. They also need to be par­tic­u­larly care­ful since an­titrust cases have busted the dom­i­nance of sev­eral mas­sive tech com­pa­nies be­fore. (While they could swing a few elec­tions be­fore be­ing heav­ily pun­ished, I don’t think this is unique to the in­ter­net age—a small ca­bal of news­pa­per own­ers could prob­a­bly have done the same cen­turies ago). Similarly, I think the founders of Wikipe­dia ac­tu­ally had fairly lit­tle coun­ter­fac­tual im­pact, and cur­rently have fairly lit­tle power, be­cause they’re re­li­ant on ed­i­tors who are com­mit­ted to im­par­tial­ity.

What we should be more in­ter­ested in is cases where small groups didn’t just ride a trend, but ac­tu­ally cre­ated or sig­nifi­cantly boosted it. Even in those cases, though, there’s a big differ­ence be­tween suc­cess and im­pact. Lots of peo­ple have be­come very rich from shuffling around fi­nan­cial prod­ucts or ad space in novel ways. But if we look at the last fifty years over­all, they’re far from dom­i­nated by ex­treme trans­for­ma­tive events—in fact, Western so­cieties have changed very lit­tle in most ways. Apart from IT, our tech­nol­ogy re­mains roughly the same, our phys­i­cal sur­round­ings are pretty similar, and our stan­dards of liv­ing have stayed flat or even dropped slightly. (This is a ver­sion of Tyler Cowen and Peter Thiel’s views; for a bet­ter ar­tic­u­la­tion, I recom­mend The Great Stag­na­tion or The Com­pla­cent Class). Well, isn’t IT enough to make up for that? I think it will be even­tu­ally, as AI de­vel­ops, but right now most of the time spent on the in­ter­net is wasted. I don’t think cur­rent IT has had much of an effect by stan­dard met­rics of labour pro­duc­tivity, for ex­am­ple.

Should you pivot?

Ben might claim that this is be­cause few peo­ple have been op­ti­mis­ing hard for pos­i­tive im­pact us­ing high-var­i­ance strate­gies. While I agree to some ex­tent, I also think that there are pretty strong in­cen­tives to have im­pact re­gard­less. We’re in the sort of startup econ­omy where scale comes first and mon­eti­sa­tion comes sec­ond, and so en­trepreneurs already strive to cre­ate prod­ucts which in­fluence mil­lions of peo­ple even when there’s no clear way to profit from them. And en­trepreneurs are definitely no strangers to high-var­i­ance strate­gies, so I ex­pect most ap­proaches to large-scale in­fluence to already have been tried.

On the other hand, I do think that re­duc­ing ex­is­ten­tial risk is an area where a small group of peo­ple are man­ag­ing to have a large in­fluence, a claim which seems to con­trast with the as­ser­tion above. I’m not en­tirely sure how to re­solve this ten­sion, but I’ve been think­ing lately about an anal­ogy from fi­nance. Here’s Tyler Cowen:

I see a lot of money man­agers, so there’s Ray Dalio at Bridge­wa­ter. He saw one ba­sic point about real in­ter­est rates, made billions off of that over a great run. Now it’s not ob­vi­ous he and his team knew any bet­ter than any­one else.
Peter Lynch, he had fan­tas­tic in­sights into con­sumer prod­ucts. Use stuff, see how you like it, buy that stock. He be­lieved that in an age when con­sumer product stocks were tak­ing off.
War­ren Buffett, a cer­tain kind of value in­vest­ing. Worked great for a while, no big suc­cess, a lot of big failures in re­cent times.

The anal­ogy isn’t perfect, but the idea I want to ex­tract is some­thing like: once you’ve iden­ti­fied a win­ning strat­egy or idea, you can achieve great things by ex­ploit­ing it—but this shouldn’t be taken as strong ev­i­dence that you can do ex­cep­tional things in gen­eral. For ex­am­ple, hav­ing a cer­tain type of per­son­al­ity and be­ing a fan of sci­ence fic­tion is very use­ful in iden­ti­fy­ing x-risk as a pri­or­ity, but not very use­ful in found­ing a suc­cess­ful startup. Similarly, be­ing a philoso­pher is very use­ful in iden­ti­fy­ing that helping the global poor is morally im­por­tant, but not very use­ful in figur­ing out how to solve sys­temic poverty.

From this mind­set, in­stead of look­ing for big wins like “im­prov­ing in­tel­lec­tual co­or­di­na­tion”, we should be look­ing for things which are easy con­di­tional on ex­is­ten­tial risk ac­tu­ally be­ing im­por­tant, and con­di­tional on the par­tic­u­lar skil­lsets of x-risk re­duc­tion ad­vo­cates. Another way of think­ing about this is as a dis­tinc­tion be­tween high-im­pact goals and high-var­i­ance strate­gies: once you’ve iden­ti­fied a high-im­pact goal, you can pur­sue it with­out us­ing high-var­i­ance strate­gies. Startup X may have a crazy new busi­ness idea, but they prob­a­bly shouldn’t ex­e­cute it in crazy new ways. Ac­tu­ally, their best bet is likely to be join­ing Y Com­bi­na­tor, get­ting a bunch of VC fund­ing, and fol­low­ing Paul Gra­ham’s stan­dard ad­vice. Similarly, re­duc­ing x-risk is a crazy new idea for how to im­prove the world, but it’s pretty plau­si­ble that we should pur­sue it in ways similar to those which other suc­cess­ful move­ments used. Here are some stan­dard things that have his­tor­i­cally been very helpful for chang­ing the world:

  • ded­i­cated activists

  • good research

  • money

  • pub­lic support

  • poli­ti­cal influence

My prior says that all of these things mat­ter, and that most big wins will be due to di­rect effects on these things. The last two are the ones which we’re dis­pro­por­tionately lack­ing; I’m more op­ti­mistic about the lat­ter for a va­ri­ety of rea­sons.

AI policy is a par­tic­u­larly good place to have a large im­pact.

Here’s a gen­eral ar­gu­ment: gov­ern­ments are very big lev­ers, be­cause of their scale and abil­ity to ap­ply co­er­cion. A new law can be a black swan all by it­self. When I think of re­ally mas­sive wins over the past half-cen­tury, I think about the erad­i­ca­tion of smal­l­pox and po­lio, the de­vel­op­ment of space tech­nol­ogy, and the de­vel­op­ment of the in­ter­net. All of these re­lied on and were driven by gov­ern­ments. Then, of course, there are the mas­sive de­clines in poverty across Asia in par­tic­u­lar. It’s difficult to as­sign credit for this, since it’s so tied up with global­i­sa­tion, but to the ex­tent that any small group was re­spon­si­ble, it was Asian gov­ern­ments and the poli­cies of Deng Xiaop­ing, Lee Kuan Yew, Ra­jiv Gandhi, etc.

You might agree that gov­ern­ments do im­por­tant things, but think that in­fluenc­ing them is very difficult. Firstly, that’s true for most black swans, so I don’t think that should make policy work much less promis­ing even from Ben’s per­spec­tive. But sec­ondly, from the out­side view, our chances are pretty good. We’re a move­ment com­pris­ing many very com­pe­tent, clever and com­mit­ted peo­ple. We’ve got the sort of back­ing that makes poli­cy­mak­ers take peo­ple se­ri­ously: we’re af­fili­ated with lead­ing uni­ver­si­ties, tech com­pa­nies, and pub­lic figures. It’s likely that a num­ber of EAs at the best uni­ver­si­ties already have friends who will end up in top gov­ern­ment po­si­tions. We have enough money to do ex­ten­sive lob­by­ing, if that’s judged a good idea. Also, we’re cor­rect, which usu­ally helps. The main ad­van­tage we’re miss­ing is wide­spread pop­u­lar sup­port, but I don’t model this as be­ing cru­cial for is­sues where what’s needed is tar­geted in­ter­ven­tions which “pull the rope side­ways”. (We’re also miss­ing knowl­edge about what those in­ter­ven­tions should be, but that makes policy re­search even more valuable).

Here’s a more spe­cific route to im­pact: in a few decades (as­sum­ing long timelines and slow take­off) AIs that are less gen­er­ally in­tel­li­gent that hu­mans will be caus­ing poli­ti­cal and eco­nomic shock­waves, whether that’s via mass un­em­ploy­ment, en­abling large-scale se­cu­rity breaches, de­sign­ing more de­struc­tive weapons, psy­cholog­i­cal ma­nipu­la­tion, or some­thing even less pre­dictable. At this point, gov­ern­ments will panic and AI policy ad­vi­sors will have real in­fluence. If com­pe­tent and al­igned peo­ple were the ob­vi­ous choice for those po­si­tions, that’d be fan­tas­tic. If those peo­ple had spent sev­eral decades re­search­ing what in­ter­ven­tions would be most valuable, that’d be even bet­ter.

This per­spec­tive is in­spired by Mil­ton Fried­man, who ar­gued that the way to cre­ate large-scale change is by nur­tur­ing ideas which will be seized upon in a crisis.

Only a crisis—ac­tual or per­ceived—pro­duces real change. When that crisis oc­curs, the ac­tions that are taken de­pend on the ideas that are ly­ing around. That, I be­lieve, is our ba­sic func­tion: to de­velop al­ter­na­tives to ex­ist­ing poli­cies, to keep them al­ive and available un­til the poli­ti­cally im­pos­si­ble be­comes the pos­si­ble.

The ma­jor in­fluence of the In­sti­tute of Eco­nomic Af­fairs on Thatcher’s poli­cies is an ex­am­ple of this strat­egy’s suc­cess. An ad­van­tage of this ap­proach is that it can be im­ple­mented by clus­ter­ings of like-minded peo­ple col­lab­o­rat­ing with each other; for that rea­son, I’m not so wor­ried about policy work cul­ti­vat­ing the wrong mind­set (I’d be more wor­ried on this front if policy re­searchers were very widely spread out).

Another fairly spe­cific route to im­pact: sev­eral ma­jor AI re­search labs would likely act on sug­ges­tions for co­or­di­nat­ing to make AI safer, if we had any. Right now I don’t think we do, and so re­search into that could have a big mul­ti­plier. If a gov­ern­ment ends up run­ning a ma­jor AI lab (which seems pretty likely con­di­tional on long timelines) then they may also end up fol­low­ing this ad­vice, via the effect de­scribed in the para­graph above.

Un­der­ly­ing gen­er­a­tors of this disagreement

More gen­er­ally, Ben and I dis­agree on where the bot­tle­neck to AI safety is. I think that find­ing a tech­ni­cal solu­tion is prob­a­ble, but that most solu­tions would still re­quire care­ful over­sight, which may or may not hap­pen (maybe 50-50). Ben thinks that find­ing a tech­ni­cal solu­tion is im­prob­a­ble, but that if it’s found it’ll prob­a­bly be im­ple­mented well. I also have more cre­dence on long timelines and slow take­offs than he does. I think that these dis­agree­ments af­fect our views on the im­por­tance of in­fluenc­ing gov­ern­ments in par­tic­u­lar.

We also have differ­ing views on what the x-risk re­duc­tion com­mu­nity should look like. I favour a broader, more di­verse com­mu­nity; Ben favours a nar­rower, more com­mit­ted com­mu­nity. I don’t want to dis­cuss this ex­ten­sively here, but I will point out that there are many peo­ple who are much bet­ter at work­ing within a sys­tem than out­side it—peo­ple who would do well in AI safety PhDs, but couldn’t just teach them­selves to do good re­search from scratch like Nate Soares did; brilli­ant yet ab­sent-minded math­e­mat­i­ci­ans; peo­ple who could run an ex­cel­lent policy re­search group but not an ex­cel­lent startup. I think it’s valuable for such peo­ple (amongst which I in­clude my­self), to have a “de­fault” path to im­pact, even at the cost of re­duc­ing the pres­sure to be en­trepreneurial or agenty. I think this is pretty un­de­ni­able when it comes to tech­ni­cal re­search, and cross-ap­plies straight­for­wardly to policy re­search and ad­vo­cacy.

Ben and I agree that go­ing into policy is much more valuable if you’re think­ing very strate­gi­cally and out of the “out of the box” box than if you’re not. Given this mind­set, there will prob­a­bly turn out to be valuable non-stan­dard things which you can do.

Do note that this es­say is in­trin­si­cally skewed since I haven’t por­trayed Ben’s ar­gu­ments in full fidelity and have spent many more words ar­gu­ing my side. Also note that, de­spite be­ing skep­ti­cal about some of Ben’s points, I think his over­all view is im­por­tant and in­ter­est­ing and more peo­ple should be think­ing along similar lines.

Thanks to An­jali Gopal for com­ments on drafts.