What failure looks like

[paulfchristiano]

The stereo­typed image of AI catas­tro­phe is a pow­er­ful, mal­i­cious AI sys­tem that takes its cre­ators by sur­prise and quickly achieves a de­ci­sive ad­van­tage over the rest of hu­man­ity.

I think this is prob­a­bly not what failure will look like, and I want to try to paint a more re­al­is­tic pic­ture. I’ll tell the story in two parts:

• Part I: ma­chine learn­ing will in­crease our abil­ity to “get what we can mea­sure,” which could cause a slow-rol­ling catas­tro­phe. (“Go­ing out with a whim­per.”)

• Part II: ML train­ing, like com­pet­i­tive economies or nat­u­ral ecosys­tems, can give rise to “greedy” pat­terns that try to ex­pand their own in­fluence. Such pat­terns can ul­ti­mately dom­i­nate the be­hav­ior of a sys­tem and cause sud­den break­downs. (“Go­ing out with a bang,” an in­stance of op­ti­miza­tion dae­mons.)

I think these are the most im­por­tant prob­lems if we fail to solve in­tent al­ign­ment.

In prac­tice these prob­lems will in­ter­act with each other, and with other dis­rup­tions/​in­sta­bil­ity caused by rapid progress. Th­ese prob­lems are worse in wor­lds where progress is rel­a­tively fast, and fast take­off can be a key risk fac­tor, but I’m scared even if we have sev­eral years.

With fast enough take­off, my ex­pec­ta­tions start to look more like the car­i­ca­ture—this post en­vi­sions rea­son­ably broad de­ploy­ment of AI, which be­comes less and less likely as things get faster. I think the ba­sic prob­lems are still es­sen­tially the same though, just oc­cur­ring within an AI lab rather than across the world.

(None of the con­cerns in this post are novel.)

Part I: You get what you measure

If I want to con­vince Bob to vote for Alice, I can ex­per­i­ment with many differ­ent per­sua­sion strate­gies and see which ones work. Or I can build good pre­dic­tive mod­els of Bob’s be­hav­ior and then search for ac­tions that will lead him to vote for Alice. Th­ese are pow­er­ful tech­niques for achiev­ing any goal that can be eas­ily mea­sured over short time pe­ri­ods.

But if I want to help Bob figure out whether he should vote for Alice—whether vot­ing for Alice would ul­ti­mately help cre­ate the kind of so­ciety he wants—that can’t be done by trial and er­ror. To solve such tasks we need to un­der­stand what we are do­ing and why it will yield good out­comes. We still need to use data in or­der to im­prove over time, but we need to un­der­stand how to up­date on new data in or­der to im­prove.

Some ex­am­ples of easy-to-mea­sure vs. hard-to-mea­sure goals:

• Per­suad­ing me, vs. helping me figure out what’s true. (Thanks to Wei Dai for mak­ing this ex­am­ple crisp.)

• Re­duc­ing my feel­ing of un­cer­tainty, vs. in­creas­ing my knowl­edge about the world.

• Im­prov­ing my re­ported life satis­fac­tion, vs. ac­tu­ally helping me live a good life.

• Re­duc­ing re­ported crimes, vs. ac­tu­ally pre­vent­ing crime.

• In­creas­ing my wealth on pa­per, vs. in­creas­ing my effec­tive con­trol over re­sources.

It’s already much eas­ier to pur­sue easy-to-mea­sure goals, but ma­chine learn­ing will widen the gap by let­ting us try a huge num­ber of pos­si­ble strate­gies and search over mas­sive spaces of pos­si­ble ac­tions. That force will com­bine with and am­plify ex­ist­ing in­sti­tu­tional and so­cial dy­nam­ics that already fa­vor eas­ily-mea­sured goals.

Right now hu­mans think­ing and talk­ing about the fu­ture they want to cre­ate are a pow­er­ful force that is able to steer our tra­jec­tory. But over time hu­man rea­son­ing will be­come weaker and weaker com­pared to new forms of rea­son­ing honed by trial-and-er­ror. Even­tu­ally our so­ciety’s tra­jec­tory will be de­ter­mined by pow­er­ful op­ti­miza­tion with eas­ily-mea­surable goals rather than by hu­man in­ten­tions about the fu­ture.

We will try to har­ness this power by con­struct­ing prox­ies for what we care about, but over time those prox­ies will come apart:

• Cor­po­ra­tions will de­liver value to con­sumers as mea­sured by profit. Even­tu­ally this mostly means ma­nipu­lat­ing con­sumers, cap­tur­ing reg­u­la­tors, ex­tor­tion and theft.

• In­vestors will “own” shares of in­creas­ingly prof­itable cor­po­ra­tions, and will some­times try to use their prof­its to af­fect the world. Even­tu­ally in­stead of ac­tu­ally hav­ing an im­pact they will be sur­rounded by ad­vi­sors who ma­nipu­late them into think­ing they’ve had an im­pact.

• Law en­force­ment will drive down com­plaints and in­crease re­ported sense of se­cu­rity. Even­tu­ally this will be driven by cre­at­ing a false sense of se­cu­rity, hid­ing in­for­ma­tion about law en­force­ment failures, sup­press­ing com­plaints, and co­erc­ing and ma­nipu­lat­ing cit­i­zens.

• Leg­is­la­tion may be op­ti­mized to seem like it is ad­dress­ing real prob­lems and helping con­stituents. Even­tu­ally that will be achieved by un­der­min­ing our abil­ity to ac­tu­ally per­ceive prob­lems and con­struct­ing in­creas­ingly con­vinc­ing nar­ra­tives about where the world is go­ing and what’s im­por­tant.

For a while we will be able to over­come these prob­lems by rec­og­niz­ing them, im­prov­ing the prox­ies, and im­pos­ing ad-hoc re­stric­tions that avoid ma­nipu­la­tion or abuse. But as the sys­tem be­comes more com­plex, that job it­self be­comes too challeng­ing for hu­man rea­son­ing to solve di­rectly and re­quires its own trial and er­ror, and at the meta-level the pro­cess con­tinues to pur­sue some eas­ily mea­sured ob­jec­tive (po­ten­tially over longer timescales). Even­tu­ally large-scale at­tempts to fix the prob­lem are them­selves op­posed by the col­lec­tive op­ti­miza­tion of mil­lions of op­ti­miz­ers pur­su­ing sim­ple goals.

As this world goes off the rails, there may not be any dis­crete point where con­sen­sus rec­og­nizes that things have gone off the rails.

Amongst the broader pop­u­la­tion, many folk already have a vague pic­ture of the over­all tra­jec­tory of the world and a vague sense that some­thing has gone wrong. There may be sig­nifi­cant pop­ulist pushes for re­form, but in gen­eral these won’t be well-di­rected. Some states may re­ally put on the brakes, but they will rapidly fall be­hind eco­nom­i­cally and mil­i­tar­ily, and in­deed “ap­pear to be pros­per­ous” is one of the eas­ily-mea­sured goals for which the in­com­pre­hen­si­ble sys­tem is op­ti­miz­ing.

Amongst in­tel­lec­tual elites there will be gen­uine am­bi­guity and un­cer­tainty about whether the cur­rent state of af­fairs is good or bad. Peo­ple re­ally will be get­ting richer for a while. Over the short term, the forces grad­u­ally wrest­ing con­trol from hu­mans do not look so differ­ent from (e.g.) cor­po­rate lob­by­ing against the pub­lic in­ter­est, or prin­ci­pal-agent prob­lems in hu­man in­sti­tu­tions. There will be le­gi­t­i­mate ar­gu­ments about whether the im­plicit long-term pur­poses be­ing pur­sued by AI sys­tems are re­ally so much worse than the long-term pur­poses that would be pur­sued by the share­hold­ers of pub­lic com­pa­nies or cor­rupt offi­cials.

We might de­scribe the re­sult as “go­ing out with a whim­per.” Hu­man rea­son­ing grad­u­ally stops be­ing able to com­pete with so­phis­ti­cated, sys­tem­atized ma­nipu­la­tion and de­cep­tion which is con­tin­u­ously im­prov­ing by trial and er­ror; hu­man con­trol over lev­ers of power grad­u­ally be­comes less and less effec­tive; we ul­ti­mately lose any real abil­ity to in­fluence our so­ciety’s tra­jec­tory. By the time we spread through the stars our cur­rent val­ues are just one of many forces in the world, not even a par­tic­u­larly strong one.

Part II: in­fluence-seek­ing be­hav­ior is scary

There are some pos­si­ble pat­terns that want to seek and ex­pand their own in­fluence—or­ganisms, cor­rupt bu­reau­crats, com­pa­nies ob­sessed with growth. If such pat­terns ap­pear, they will tend to in­crease their own in­fluence and so can come to dom­i­nate the be­hav­ior of large com­plex sys­tems un­less there is com­pe­ti­tion or a suc­cess­ful effort to sup­press them.

Modern ML in­stan­ti­ates mas­sive num­bers of cog­ni­tive poli­cies, and then fur­ther re­fines (and ul­ti­mately de­ploys) what­ever poli­cies perform well ac­cord­ing to some train­ing ob­jec­tive. If progress con­tinues, even­tu­ally ma­chine learn­ing will prob­a­bly pro­duce sys­tems that have a de­tailed un­der­stand­ing of the world, which are able to adapt their be­hav­ior in or­der to achieve spe­cific goals.

Once we start search­ing over poli­cies that un­der­stand the world well enough, we run into a prob­lem: any in­fluence-seek­ing poli­cies we stum­ble across would also score well ac­cord­ing to our train­ing ob­jec­tive, be­cause perform­ing well on the train­ing ob­jec­tive is a good strat­egy for ob­tain­ing in­fluence.

How fre­quently will we run into in­fluence-seek­ing poli­cies, vs. poli­cies that just straight­for­wardly pur­sue the goals we wanted them to? I don’t know.

One rea­son to be scared is that a wide va­ri­ety of goals could lead to in­fluence-seek­ing be­hav­ior, while the “in­tended” goal of a sys­tem is a nar­rower tar­get, so we might ex­pect in­fluence-seek­ing be­hav­ior to be more com­mon in the broader land­scape of “pos­si­ble cog­ni­tive poli­cies.”

One rea­son to be re­as­sured is that we perform this search by grad­u­ally mod­ify­ing suc­cess­ful poli­cies, so we might ob­tain poli­cies that are roughly do­ing the right thing at an early enough stage that “in­fluence-seek­ing be­hav­ior” wouldn’t ac­tu­ally be so­phis­ti­cated enough to yield good train­ing perfor­mance. On the other hand, even­tu­ally we’d en­counter sys­tems that did have that level of so­phis­ti­ca­tion, and if they didn’t yet have a perfect con­cep­tion of the goal then “slightly in­crease their de­gree of in­fluence-seek­ing be­hav­ior” would be just as good a mod­ifi­ca­tion as “slightly im­prove their con­cep­tion of the goal.”

Over­all it seems very plau­si­ble to me that we’d en­counter in­fluence-seek­ing be­hav­ior “by de­fault,” and pos­si­ble (though less likely) that we’d get it al­most all of the time even if we made a re­ally con­certed effort to bias the search to­wards “straight­for­wardly do what we want.”

If such in­fluence-seek­ing be­hav­ior emerged and sur­vived the train­ing pro­cess, then it could quickly be­come ex­tremely difficult to root out. If you try to al­lo­cate more in­fluence to sys­tems that seem nice and straight­for­ward, you just en­sure that “seem nice and straight­for­ward” is the best strat­egy for seek­ing in­fluence. Un­less you are re­ally care­ful about test­ing for “seem nice” you can make things even worse, since an in­fluence-seeker would be ag­gres­sively gam­ing what­ever stan­dard you ap­plied. And as the world be­comes more com­plex, there are more and more op­por­tu­ni­ties for in­fluence-seek­ers to find other chan­nels to in­crease their own in­fluence.

At­tempts to sup­press in­fluence-seek­ing be­hav­ior (call them “im­mune sys­tems”) rest on the sup­pres­sor hav­ing some kind of epistemic ad­van­tage over the in­fluence-seeker. Once the in­fluence-seek­ers can out­think an im­mune sys­tem, they can avoid de­tec­tion and po­ten­tially even com­pro­mise the im­mune sys­tem to fur­ther ex­pand their in­fluence. If ML sys­tems are more so­phis­ti­cated than hu­mans, im­mune sys­tems must them­selves be au­to­mated. And if ML plays a large role in that au­toma­tion, then the im­mune sys­tem is sub­ject to the same pres­sure to­wards in­fluence-seek­ing.

This con­cern doesn’t rest on a de­tailed story about mod­ern ML train­ing. The im­por­tant fea­ture is that we in­stan­ti­ate lots of pat­terns that cap­ture so­phis­ti­cated rea­son­ing about the world, some of which may be in­fluence-seek­ing. The con­cern ex­ists whether that rea­son­ing oc­curs within a sin­gle com­puter, or is im­ple­mented in a messy dis­tributed way by a whole econ­omy of in­ter­act­ing agents—whether trial and er­ror takes the form of gra­di­ent de­scent or ex­plicit tweak­ing and op­ti­miza­tion by en­g­ineers try­ing to de­sign a bet­ter au­to­mated com­pany. Avoid­ing end-to-end op­ti­miza­tion may help pre­vent the emer­gence of in­fluence-seek­ing be­hav­iors (by im­prov­ing hu­man un­der­stand­ing of and hence con­trol over the kind of rea­son­ing that emerges). But once such pat­terns ex­ist a messy dis­tributed world just cre­ates more and more op­por­tu­ni­ties for in­fluence-seek­ing pat­terns to ex­pand their in­fluence.

If in­fluence-seek­ing pat­terns do ap­pear and be­come en­trenched, it can ul­ti­mately lead to a rapid phase tran­si­tion from the world de­scribed in Part I to a much worse situ­a­tion where hu­mans to­tally lose con­trol.

Early in the tra­jec­tory, in­fluence-seek­ing sys­tems mostly ac­quire in­fluence by mak­ing them­selves use­ful and look­ing as in­nocu­ous as pos­si­ble. They may provide use­ful ser­vices in the econ­omy in or­der to make money for them and their own­ers, make ap­par­ently-rea­son­able policy recom­men­da­tions in or­der to be more widely con­sulted for ad­vice, try to help peo­ple feel happy, etc. (This world is still plagued by the prob­lems in part I.)

From time to time AI sys­tems may fail catas­troph­i­cally. For ex­am­ple, an au­to­mated cor­po­ra­tion may just take the money and run; a law en­force­ment sys­tem may abruptly start seiz­ing re­sources and try­ing to defend it­self from at­tempted de­com­mis­sion when the bad be­hav­ior is de­tected; etc. Th­ese prob­lems may be con­tin­u­ous with some of the failures dis­cussed in Part I—there isn’t a clean line be­tween cases where a proxy breaks down com­pletely, and cases where the sys­tem isn’t even pur­su­ing the proxy.

There will likely be a gen­eral un­der­stand­ing of this dy­namic, but it’s hard to re­ally pin down the level of sys­temic risk and miti­ga­tion may be ex­pen­sive if we don’t have a good tech­nolog­i­cal solu­tion. So we may not be able to muster up a re­sponse un­til we have a clear warn­ing shot—and if we do well about nip­ping small failures in the bud, we may not get any medium-sized warn­ing shots at all.

Even­tu­ally we reach the point where we could not re­cover from a cor­re­lated au­toma­tion failure. Un­der these con­di­tions in­fluence-seek­ing sys­tems stop be­hav­ing in the in­tended way, since their in­cen­tives have changed—they are now more in­ter­ested in con­trol­ling in­fluence af­ter the re­sult­ing catas­tro­phe then con­tin­u­ing to play nice with ex­ist­ing in­sti­tu­tions and in­cen­tives.

An un­re­cov­er­able catas­tro­phe would prob­a­bly oc­cur dur­ing some pe­riod of height­ened vuln­er­a­bil­ity—a con­flict be­tween states, a nat­u­ral dis­aster, a se­ri­ous cy­ber­at­tack, etc.---since that would be the first mo­ment that re­cov­ery is im­pos­si­ble and would cre­ate lo­cal shocks that could pre­cip­i­tate catas­tro­phe. The catas­tro­phe might look like a rapidly cas­cad­ing se­ries of au­toma­tion failures: A few au­to­mated sys­tems go off the rails in re­sponse to some lo­cal shock. As those sys­tems go off the rails, the lo­cal shock is com­pounded into a larger dis­tur­bance; more and more au­to­mated sys­tems move fur­ther from their train­ing dis­tri­bu­tion and start failing. Real­is­ti­cally this would prob­a­bly be com­pounded by wide­spread hu­man failures in re­sponse to fear and break­down of ex­ist­ing in­cen­tive sys­tems—many things start break­ing as you move off dis­tri­bu­tion, not just ML.

It is hard to see how un­aided hu­mans could re­main ro­bust to this kind of failure with­out an ex­plicit large-scale effort to re­duce our de­pen­dence on po­ten­tially brit­tle ma­chines, which might it­self be very ex­pen­sive.

I’d de­scribe this re­sult as “go­ing out with a bang.” It prob­a­bly re­sults in lots of ob­vi­ous de­struc­tion, and it leaves us no op­por­tu­nity to course-cor­rect af­ter­wards. In terms of im­me­di­ate con­se­quences it may not be eas­ily dis­t­in­guished from other kinds of break­down of com­plex /​ brit­tle /​ co-adapted sys­tems, or from con­flict (since there are likely to be many hu­mans who are sym­pa­thetic to AI sys­tems). From my per­spec­tive the key differ­ence be­tween this sce­nario and nor­mal ac­ci­dents or con­flict is that af­ter­wards we are left with a bunch of pow­er­ful in­fluence-seek­ing sys­tems, which are so­phis­ti­cated enough that we can prob­a­bly not get rid of them.

It’s also pos­si­ble to meet a similar fate re­sult with­out any overt catas­tro­phe (if we last long enough). As law en­force­ment, gov­ern­ment bu­reau­cra­cies, and mil­i­taries be­come more au­to­mated, hu­man con­trol be­comes in­creas­ingly de­pen­dent on a com­pli­cated sys­tem with lots of mov­ing parts. One day lead­ers may find that de­spite their nom­i­nal au­thor­ity they don’t ac­tu­ally have con­trol over what these in­sti­tu­tions do. For ex­am­ple, mil­i­tary lead­ers might is­sue an or­der and find it is ig­nored. This might im­me­di­ately prompt panic and a strong re­sponse, but the re­sponse it­self may run into the same prob­lem, and at that point the game may be up.

Similar blood­less rev­olu­tions are pos­si­ble if in­fluence-seek­ers op­er­ate legally, or by ma­nipu­la­tion and de­cep­tion, or so on. Any pre­cise vi­sion for catas­tro­phe will nec­es­sar­ily be highly un­likely. But if in­fluence-seek­ers are rou­tinely in­tro­duced by pow­er­ful ML and we are not able to se­lect against them, then it seems like things won’t go well.

[Wei_Dai]

I think AI risk is dis­junc­tive enough that it’s not clear most of the prob­a­bil­ity mass can be cap­tured by a sin­gle sce­nario/​story, even as broad as this one tries to be. Here are some ad­di­tional sce­nar­ios that don’t fit into this story or aren’t made very salient by it.

1. AI-pow­ered memetic war­fare makes all hu­mans effec­tively in­sane.

2. Hu­mans break off into var­i­ous groups to colonize the uni­verse with the help of their AIs. Due to in­suffi­cient “metaphilo­soph­i­cal pa­ter­nal­ism”, they each con­struct their own ver­sion of utopia which is ei­ther di­rectly bad (i.e., some of the “utopias” are ob­jec­tively ter­rible or sub­jec­tively ter­rible ac­cord­ing to my val­ues), or bad be­cause of op­por­tu­nity costs.

3. AI-pow­ered economies have much higher economies of scale be­cause AIs don’t suffer from the kind of co­or­di­na­tion costs that hu­mans have (e.g., they can merge their util­ity func­tions and be­come clones of each other). Some coun­tries may try to pre­vent AI-man­aged com­pa­nies from merg­ing for ide­olog­i­cal or safety rea­sons, but oth­ers (in or­der to gain a com­pet­i­tive ad­van­tage on the world stage) will ba­si­cally al­low their whole econ­omy to be con­trol­led by one AI, which even­tu­ally achieves a de­ci­sive ad­van­tage over the rest of hu­man­ity and does a treach­er­ous turn.

4. The same in­cen­tive for AIs to merge might also cre­ate an in­cen­tive for value lock-in, in or­der to fa­cil­i­tate the merg­ing. (AIs that don’t have util­ity func­tions might have a harder time co­or­di­nat­ing with each other.) Other in­cen­tives for pre­ma­ture value lock-in might in­clude defense against value ma­nipu­la­tion/​cor­rup­tion/​drift. So AIs end up em­body­ing locked-in ver­sions of hu­man val­ues which are ter­rible in light of our true/​ac­tual val­ues.

5. I think the origi­nal “stereo­typed image of AI catas­tro­phe” is still quite plau­si­ble, if for ex­am­ple there is a large amount of hard­ware over­hang be­fore the last piece of puz­zle for build­ing AGI falls into place.

[paulfchristiano]

I think of #3 and #5 as risk fac­tors that com­pound the risks I’m de­scribing—they are two (of many!) ways that the de­tailed pic­ture could look differ­ent, but don’t change the broad out­line. I think it’s par­tic­u­larly im­por­tant to un­der­stand what failure looks like un­der a more “busi­ness as usual” sce­nario, so that peo­ple can sep­a­rate ob­jec­tions to the ex­is­tence of any risk from ob­jec­tions to other ex­ac­er­bat­ing fac­tors that we are con­cerned about (like fast take­off, war, peo­ple be­ing asleep at the wheel, etc.)

I’d clas­sify #1, #2, and #4 as differ­ent prob­lems not re­lated to in­tent al­ign­ment per se (though in­tent al­ign­ment may let us build AI sys­tems that can help ad­dress these prob­lems). I think the more gen­eral point is: if you think AI progress is likely to drive many of the biggest up­com­ing changes in the world, then there will be lots of risks as­so­ci­ated with AI. Here I’m just try­ing to clar­ify what hap­pens if we fail to solve in­tent al­ign­ment.

[Wei_Dai]

I’m not sure I un­der­stand the dis­tinc­tion you’re draw­ing be­tween risk fac­tors that com­pound the risks that you’re de­scribing vs. differ­ent prob­lems not re­lated to in­tent al­ign­ment per se. It seems to me like “AI-pow­ered economies have much higher economies of scale be­cause AIs don’t suffer from the kind of co­or­di­na­tion costs that hu­mans have (e.g., they can merge their util­ity func­tions and be­come clones of each other)” is a sep­a­rate prob­lem from solv­ing in­tent al­ign­ment, whereas “AI-pow­ered memetic war­fare makes all hu­mans effec­tively in­sane” is kind of an ex­treme case of “ma­chine learn­ing will in­crease our abil­ity to ‘get what we can mea­sure’” which seems to be the op­po­site of how you clas­sify them.

What do you think are the im­pli­ca­tions of some­thing be­long­ing to one cat­e­gory ver­sus an­other (i.e., is there some­thing we should do differ­ently de­pend­ing on which of these cat­e­gories a risk fac­tor /​ prob­lem be­longs to)?

I think the more gen­eral point is: if you think AI progress is likely to drive many of the biggest up­com­ing changes in the world, then there will be lots of risks as­so­ci­ated with AI. Here I’m just try­ing to clar­ify what hap­pens if we fail to solve in­tent al­ign­ment.

Ah, when I read “I think this is prob­a­bly not what failure will look like” I in­ter­preted that to mean “failure to pre­vent AI risk”, and then I missed the clar­ifi­ca­tion “these are the most im­por­tant prob­lems if we fail to solve in­tent al­ign­ment” that came later in the post, in part be­cause of a bug in GW that caused the post to be in­cor­rectly for­mat­ted.

Aside from that, I’m wor­ried about tel­ling a vivid story about one par­tic­u­lar AI risk, un­less you re­ally ham­mer the point that it’s just one risk out of many, oth­er­wise it seems too easy for the reader to get that story stuck in their mind and come to think that this is the main or only thing they have to worry about as far as AI is con­cerned.

[CarlShulman]

I think the kind of phras­ing you use in this post and oth­ers like it sys­tem­at­i­cally mis­leads read­ers into think­ing that in your sce­nar­ios there are no robot armies seiz­ing con­trol of the world (or rather, that all armies worth any­thing at that point are robotic, and so AIs in con­flict with hu­man­ity means mil­i­tary force that hu­man­ity can­not over­come). I.e. AI sys­tems pur­su­ing badly al­igned proxy goals or in­fluence-seek­ing ten­den­cies wind up con­trol­ling or cre­at­ing that mil­i­tary power and ex­pro­pri­at­ing hu­man­ity (which even­tu­ally couldn’t fight back there­after even if unified).

E.g. Dy­lan Matthews’ Vox writeup of the OP seems to think that your sce­nar­ios don’t in­volve robot armies tak­ing con­trol of the means of pro­duc­tion and us­ing the uni­verse for their ends against hu­man ob­jec­tions or kil­ling off ex­ist­ing hu­mans (per­haps de­struc­tively scan­ning their brains for in­for­ma­tion but not giv­ing good liv­ing con­di­tions to the scanned data):

Even so, Chris­ti­ano’s first sce­nario doesn’t pre­cisely en­vi­sion hu­man ex­tinc­tion. It en­vi­sions hu­man ir­rele­vance, as we be­come agents of ma­chines we cre­ated.

Hu­man re­li­ance on these sys­tems, com­bined with the sys­tems failing, leads to a mas­sive so­cietal break­down. And in the wake of the break­down, there are still ma­chines that are great at per­suad­ing and in­fluenc­ing peo­ple to do what they want, ma­chines that got ev­ery­one into this catas­tro­phe and yet are still giv­ing ad­vice that some of us will listen to.

The Vox ar­ti­cle also mis­takes the source of in­fluence-seek­ing pat­terns to be about so­cial in­fluence rather than sys­tems that try to in­crease in power and num­bers tend to do so, so are se­lected for if we ac­ci­den­tally or in­ten­tion­ally pro­duce them and don’t effec­tively weed them out; this is why liv­ing things are adapted to sur­vive and ex­pand; such de­sires mo­ti­vate con­flict with hu­mans when power and re­pro­duc­tion can be ob­tained by con­flict with hu­mans, which can look like robot armies tak­ing con­trol.takes the point about in­fluence-seek­ing pat­terns to be about. That seems to me just a mis­take about the mean­ing of in­fluence you had in mind here:

Often, he notes, the best way to achieve a given goal is to ob­tain in­fluence over other peo­ple who can help you achieve that goal. If you are try­ing to launch a startup, you need to in­fluence in­vestors to give you money and en­g­ineers to come work for you. If you’re try­ing to pass a law, you need to in­fluence ad­vo­cacy groups and mem­bers of Congress.

That means that ma­chine-learn­ing al­gorithms will prob­a­bly, over time, pro­duce pro­grams that are ex­tremely good at in­fluenc­ing peo­ple. And it’s dan­ger­ous to have ma­chines that are ex­tremely good at in­fluenc­ing peo­ple.

[paulfchristiano]

The Vox ar­ti­cle also mis­takes the source of in­fluence-seek­ing pat­terns to be about so­cial in­fluence rather than ’sys­tems that try to in­crease in power and num­bers tend to do so, so are se­lected for if we ac­ci­den­tally or in­ten­tion­ally pro­duce them and don’t effec­tively weed them out; this is why liv­ing things are adapted to sur­vive and ex­pand; such de­sires mo­ti­vate con­flict with hu­mans when power and re­pro­duc­tion can be ob­tained by con­flict with hu­mans, which can look like robot armies tak­ing con­trol.

Yes, I agree the Vox ar­ti­cle made this mis­take. Me say­ing “in­fluence” prob­a­bly gives peo­ple the wrong idea so I should change that—I’m in­clud­ing “con­trols the mil­i­tary” as a cen­tral ex­am­ple, but it’s not what comes to mind when you hear “in­fluence.” I like “in­fluence” more than “power” be­cause it’s more spe­cific, cap­tures what we ac­tu­ally care about, and less likely to lead to a de­bate about “what is power any­way.”

In gen­eral I think the Vox ar­ti­cle’s dis­cus­sion of Part II has some prob­lems, and the dis­cus­sion of Part I is closer to the mark. (Part I is also more in line with the nar­ra­tive of the ar­ti­cle, since Part II re­ally is more like Ter­mi­na­tor. I’m not sure which way the causal­ity goes here though, i.e. whether they ended up with that nar­ra­tive based on mi­s­un­der­stand­ings about Part II or whether they framed Part II in a way that made it more con­sis­tent with the nar­ra­tive, maybe hav­ing been in­spired to write the piece based on Part I.)

There is a differ­ent mis­take with the same fla­vor, later in the Vox ar­ti­cle: “But even­tu­ally, the al­gorithms’ in­cen­tives to ex­pand in­fluence might start to over­take their in­cen­tives to achieve the speci­fied goal. That, in turn, makes the AI sys­tem worse at achiev­ing its in­tended goal, which in­creases the odds of some ter­rible failure”

The prob­lem isn’t re­ally “the AI sys­tem is worse at achiev­ing its in­tended goal;” like you say, it’s that in­fluence-seek­ing AI sys­tems will even­tu­ally be in con­flict with hu­mans, and that’s bad news if AI sys­tems are much more ca­pa­ble/​pow­er­ful than we are.

[AI sys­tems] wind up con­trol­ling or cre­at­ing that mil­i­tary power and ex­pro­pri­at­ing hu­man­ity (which couldn’t fight back there­after even if unified)

Failure would pre­sum­ably oc­cur be­fore we get to the stage of “robot army can defeat unified hu­man­ity”—failure should hap­pen soon af­ter it be­comes pos­si­ble, and there are eas­ier ways to fail than to win a clean war. Em­pha­siz­ing this may give peo­ple the wrong idea, since it makes unity and sta­bil­ity seem like a solu­tion rather than a stop­gap. But em­pha­siz­ing the robot army seems to have a similar prob­lem—it doesn’t re­ally mat­ter whether there is a literal robot army, you are in trou­ble any­way.

[CarlShulman]

Failure would pre­sum­ably oc­cur be­fore we get to the stage of “robot army can defeat unified hu­man­ity”—failure should hap­pen soon af­ter it be­comes pos­si­ble, and there are eas­ier ways to fail than to win a clean war. Em­pha­siz­ing this may give peo­ple the wrong idea, since it makes unity and sta­bil­ity seem like a solu­tion rather than a stop­gap. But em­pha­siz­ing the robot army seems to have a similar prob­lem—it doesn’t re­ally mat­ter whether there is a literal robot army, you are in trou­ble any­way.

I agree other pow­er­ful tools can achieve the same out­come, and since in prac­tice hu­man­ity isn’t unified rogue AI could act ear­lier, but ei­ther way you get to AI con­trol­ling the means of co­er­cive force, which helps peo­ple to un­der­stand the end-state reached.

It’s good to both un­der­stand the events by which one is shifted into the bad tra­jec­tory, and to be clear on what the tra­jec­tory is. It sounds like your fo­cus on the former may have in­terfered with the lat­ter.

[paulfchristiano]

I do agree there was a mis­com­mu­ni­ca­tion about the end state, and that lan­guage like “lots of ob­vi­ous de­struc­tion” is an un­der­state­ment.

I do still en­dorse “mil­i­tary lead­ers might is­sue an or­der and find it is ig­nored” (or to­tal col­lapse of so­ciety) as ba­si­cally ac­cu­rate and not an un­der­state­ment.

[paulfchristiano]

I agree that robot armies are an im­por­tant as­pect of part II.

In part I, where our only prob­lem is spec­i­fy­ing goals, I don’t ac­tu­ally think robot armies are a short-term con­cern. I think we can prob­a­bly build sys­tems that re­ally do avoid kil­ling peo­ple, e.g. by us­ing straight­for­ward ver­sions of “do things that are pre­dicted to lead to videos that peo­ple rate as ac­cept­able,” and that at the point when things have gone off the rails those videos still look fine (and to un­der­stand that there is a deep prob­lem at that point you need to en­gage with com­pli­cated facts about the situ­a­tion that are be­yond hu­man com­pre­hen­sion, not things like “are the robots kil­ling peo­ple?”). I’m not vi­su­al­iz­ing the case where no one does any­thing to try to make their AI safe, I’m imag­in­ing the most prob­a­ble cases where peo­ple fail.

I think this is an im­por­tant point, be­cause I think much dis­cus­sion of AI safety imag­ines “How can we give our AIs an ob­jec­tive which en­sures it won’t go around kil­ling ev­ery­one,” and I think that’s re­ally not the im­por­tant or in­ter­est­ing part of spec­i­fy­ing an ob­jec­tive (and so leads peo­ple to be rea­son­ably op­ti­mistic about solu­tions that I re­gard as ob­vi­ously to­tally in­ad­e­quate). I think you should only be con­cerned about your AI kil­ling ev­ery­one be­cause of in­ner al­ign­ment /​ op­ti­miza­tion dae­mons.

That said, I do ex­pect pos­si­bly-catas­trophic AI to come only shortly be­fore the sin­gu­lar­ity (in cal­en­dar time) and so the situ­a­tion “hu­mans aren’t able to steer the tra­jec­tory of so­ciety” prob­a­bly gets worse pretty quickly. I as­sume we are on the same page here.

In that sense Part I is mis­lead­ing. It de­scribes the part of the tra­jec­tory where I think the ac­tion is, the last mo­ments where we could have ac­tu­ally done some­thing to avoid doom, but from the per­spec­tive of an on­looker that pe­riod could be pretty brief. If there is a Dyson sphere in 2050 it’s not clear that any­one re­ally cares what hap­pened dur­ing 2048-2049. I think the worst offen­der is the last sen­tence of Part I (“By the time we spread through the stars...”)

Part I has this fo­cus be­cause (i) that’s where I think the ac­tion is—by the time you have robot armies kil­ling ev­ery­one the ship is so sailed, I think a rea­son­able com­mon-sense view­point would ac­knowl­edge this by re­act­ing with in­cre­dulity to the “robots kill ev­ery­one” sce­nario, and would cor­rectly place the “blame” on the point where ev­ery­thing got com­pletely out of con­trol even though there weren’t ac­tu­ally robot armies yet (ii) the al­ter­na­tive vi­su­al­iza­tion leads peo­ple to se­ri­ously un­der­es­ti­mate the difficulty of the al­ign­ment prob­lem, (iii) I was try­ing to de­scribe the part of the pic­ture which is rea­son­ably ac­cu­rate re­gard­less of my views on the sin­gu­lar­ity.

[CarlShulman]

I think we can prob­a­bly build sys­tems that re­ally do avoid kil­ling peo­ple, e.g. by us­ing straight­for­ward ver­sions of “do things that are pre­dicted to lead to videos that peo­ple rate as ac­cept­able,” and that at the point when things have gone off the rails those videos still look fine (and to un­der­stand that there is a deep prob­lem at that point you need to en­gage with com­pli­cated facts about the situ­a­tion that are be­yond hu­man com­pre­hen­sion, not things like “are the robots kil­ling peo­ple?”). I’m not vi­su­al­iz­ing the case where no one does any­thing to try to make their AI safe, I’m imag­in­ing the most prob­a­ble cases where peo­ple fail.

Haven’t you your­self writ­ten about the failure modes of ‘do things pre­dicted to lead to videos that peo­ple rate as ac­cept­able’ where the at­tack in­volves sur­rep­ti­tiously re­pro­gram­ming the cam­era to get op­ti­mal videos (in­clud­ing weird en­g­ineered videos de­signed to op­ti­mize on in­felic­i­ties in the learned ob­jec­tive? E.g. in your dis­cus­sion of an un­al­igned bench­mark here:

We train an au­tore­gres­sive gen­er­a­tive model, to max­i­mize the log prob­a­bil­ity as­signed to the ac­tual se­quence of ac­tions and ob­ser­va­tions pro­duced by the AI (with each ob­ser­va­tion con­di­tioned on the past ac­tions). This isn’t ac­tu­ally a good way to train the gen­er­a­tive model, but it’s not re­ally cen­tral to the dis­cus­sion.

We train the re­ward func­tion by show­ing hu­mans se­quences of ac­tions and pre­dicted ob­ser­va­tions, ask­ing them to as­sign scores, then pre­dict­ing those scores with su­per­vised learn­ing. We show hu­mans the se­quences of ac­tions that look most promis­ing to the sys­tem...

I am quite con­fi­dent that “world which looks ap­peal­ing on screen” is not the same as “world which is good;” the gap is already large to­day, and it will grow much larger over time.

Main­tain­ing a fic­tion. Ul­ti­mately the goal of this AI is to make the sen­sors look like good things are hap­pen­ing in the world. This can be done ei­ther by mak­ing the world good or by care­fully con­trol­ling what gets fed to the sen­sors. In gen­eral, there are many fewer con­straints on “what gets fed to the sen­sors” than “what the world is ac­tu­ally like,” and so it’s go­ing to be pos­si­ble to pro­duce much more de­sir­able-look­ing out­comes, much more eas­ily, by con­trol­ling the sen­sors.

If the hu­mans un­der­stand what the AI is do­ing in de­tail then they can ob­serve the se­quence of ac­tions to no­tice any ac­tions that might cor­rupt the sen­sors. But as soon as the AI is en­gag­ing in com­pli­cated plans this ap­pears to be to­tally in­tractable. Even if the AI was hu­man-level, there are many ex­tremely hard-to-de­tect plans: for ex­am­ple, the AI could col­lab­o­rate with an ex­ter­nal at­tacker, who will sur­rep­ti­tiously cor­rupt the sen­sor. Or the AI could cre­ate a copy of it­self on an ex­ter­nal server who then helped or­ches­trate a hard-to-de­tect cor­rup­tion.

Are you now very con­fi­dent that no AI com­pany would im­ple­ment some­thing with this vuln­er­a­bil­ity? Or does the para­graph be­low just mean that the part where the AI gets con­trol of the cam­era has to hap­pen be­fore the robot armies (or even robot armies iden­ti­fied as trace­able to the AI in ques­tion), which then hap­pen?

Part I has this fo­cus be­cause (i) that’s where I think the ac­tion is—by the time you have robot armies kil­ling ev­ery­one the ship is so sailed, I think a rea­son­able com­mon-sense view­point would ac­knowl­edge this by re­act­ing with in­cre­dulity to the “robots kill ev­ery­one” sce­nario, and would cor­rectly place the “blame” on the point where ev­ery­thing got com­pletely out of con­trol even though there weren’t ac­tu­ally robot armies yet (ii) the al­ter­na­tive vi­su­al­iza­tion leads peo­ple to se­ri­ously un­der­es­ti­mate the difficulty of the al­ign­ment prob­lem, (iii) I was try­ing to de­scribe the part of the pic­ture which is rea­son­ably ac­cu­rate re­gard­less of my views on the sin­gu­lar­ity.

Be­cause it definitely seems that Vox got the im­pres­sion from it that there is never a robot army takeover in the sce­nario, not that it’s slightly pre­ceded by cam­era hack­ing.

Is the idea that the AI sys­tems de­vel­ops goals over the ex­ter­nal world (rather than the sense in­puts/​video pix­els) so that they are re­ally pur­su­ing the ap­pear­ance of pros­per­ity, or cor­po­rate prof­its, and so don’t just wire­head their sense in­puts as in your bench­mark post?

[paulfchristiano]

My me­dian out­come is that peo­ple solve in­tent al­ign­ment well enough to avoid catas­tro­phe. Amongst the cases where we fail, my me­dian out­come is that peo­ple solve enough of al­ign­ment that they can avoid the most overt failures, like liter­ally com­pro­mis­ing sen­sors and kil­ling peo­ple (at least for a long sub­jec­tive time), and can build AIs that help defend them from other AIs. That prob­lem seems rad­i­cally eas­ier—most plau­si­ble paths to cor­rupt­ing sen­sors in­volve in­ter­me­di­ate stages with hints of cor­rup­tion that could be rec­og­nized by a weaker AI (and hence gen­er­ate low re­ward). Even­tu­ally this will break down, but it seems quite late.

very con­fi­dent that no AI com­pany would im­ple­ment some­thing with this vuln­er­a­bil­ity?

The story doesn’t de­pend on “no AI com­pany” im­ple­ment­ing some­thing that be­haves badly, it de­pends on peo­ple hav­ing ac­cess to AI that be­haves well.

Also “very con­fi­dent” seems differ­ent from “most likely failure sce­nario.”

Haven’t you your­self writ­ten about the failure modes of ‘do things pre­dicted to lead to videos that peo­ple rate as ac­cept­able’ where the at­tack in­volves sur­rep­ti­tiously re­pro­gram­ming the cam­era to get op­ti­mal videos (in­clud­ing weird en­g­ineered videos de­signed to op­ti­mize on in­felic­i­ties in the learned ob­jec­tive?

That’s a de­scrip­tion of the prob­lem /​ the be­hav­ior of the un­al­igned bench­mark, not the most likely out­come (since I think the prob­lem is most likely to be solved). We may have a differ­ence in view be­tween a dis­tri­bu­tion over out­comes that is slanted to­wards “ev­ery­thing goes well” such that the most re­al­is­tic failures are the ones that are the clos­est calls, vs. a dis­tri­bu­tion slanted to­wards “ev­ery­thing goes badly” such that the most re­al­is­tic failures are the com­plete and to­tal ones where you weren’t even close.

Be­cause it definitely seems that Vox got the im­pres­sion from it that there is never a robot army takeover in the sce­nario, not that it’s slightly pre­ceded by cam­era hack­ing.

I agree there is a robot takeover shortly later in ob­jec­tive time (mostly be­cause of the sin­gu­lar­ity). Ex­actly how long it is mostly de­pends on how early things go off the rails w.r.t. al­ign­ment, per­haps you have O(year).

[CarlShulman]

OK, thanks for the clar­ifi­ca­tion!

My own sense is that the in­ter­me­di­ate sce­nar­ios are un­sta­ble: if we have fairly al­igned AI we im­me­di­ately use it to make more al­igned AI and col­lec­tively largely re­verse things like Face­book click-max­i­miza­tion ma­nipu­la­tion. If we have lost the power to re­verse things then they go all the way to near-to­tal loss of con­trol over the fu­ture. So i would tend to think we wind up in the ex­tremes.

I could imag­ine a sce­nario where there is a close bal­ance among mul­ti­ple cen­ters of AI+hu­man power, and some but not all of those cen­ters have lo­cal AI takeovers be­fore the re­main­der solve AI al­ign­ment, and then you get a world that is a patch­work of hu­man-con­trol­led and au­tonomous states, both types au­to­mated. E.g. the United States and China are taken over by their AI sys­tems (in­lcud­ing robot armies), but the Ja­panese AI as­sis­tants and robot army re­main un­der hu­man con­trol and the fu­ture geopoli­ti­cal sys­tem keeps both types of states in­tact there­after.

[SoerenMind]

It’d be nice to hear a re­sponse from Paul to para­graph 1. My 2 cents:

I tend to agree that we end up with ex­tremes even­tu­ally. You seem to say that we would im­me­di­ately go to al­ign­ment given some­what al­igned sys­tems so Paul’s 1st story barely plays out.

Of course, the some­what al­igned sys­tems may aim at the wrong thing if we try to make them solve al­ign­ment. So the most plau­si­ble way it could work is if they pro­duce solu­tions that we can check. But if this were the case, hu­man su­per­vi­sion would be rel­a­tively easy. That’s plau­si­ble but it’s a sce­nario I care less about.

Ad­di­tion­ally, if we could use some­what al­igned sys­tems to make more al­igned ones, iter­ated am­plifi­ca­tion prob­a­bly works for al­ign­ment (nar­rowly defined by “try­ing to do what we want”). The only re­main­ing challenge would be to cre­ate one sys­tem that’s some­what smarter than us and some­what al­igned (in our case that’s true by as­sump­tion). The rest fol­lows, in­for­mally speak­ing, by in­duc­tion as long as the AI+hu­mans sys­tem can keep im­prov­ing in­tel­li­gence as al­ign­ment is im­proved. Which seems likely. That’s also plau­si­ble but it’s a big as­sump­tion and may not be the most im­por­tant sce­nario /​ isn’t a ‘tale of doom’.

[Vanessa Kosoy]

I agree that robot armies are an im­por­tant as­pect of part II.

Why? I can eas­ily imag­ine an AI takeover that works mostly through per­sua­sion/​ma­nipu­la­tion, with phys­i­cal elimi­na­tion of hu­mans com­ing only as an “af­terthought” when AI is already effec­tively in con­trol (and pro­duced ad­e­quate re­place­ments for hu­mans for the pur­pose of phys­i­cally ma­nipu­lat­ing the world). This elimi­na­tion doesn’t even re­quire an “army”, it can look like ev­ery­one agree­ing to vol­un­tary “eu­thana­sia” (pos­si­bly not un­der­stand­ing its true mean­ing). To the ex­tent phys­i­cal force is in­volved, most of it might be hu­mans against hu­mans.

[rohinmshah]

I some­what ex­pect even Part I to be solved by de­fault—it seems to rest on a premise of hu­man rea­son­ing stay­ing as pow­er­ful as it is right now, but it seems plau­si­ble that as AI sys­tems grow in ca­pa­bil­ity we will be able to lev­er­age them to im­prove hu­man rea­son­ing. Ob­vi­ously this is an ap­proach you have been push­ing, but it also seems like a nat­u­ral thing to do when you have pow­er­ful AI sys­tems.

[Zvi]

Is this fu­ture AI catas­tro­phe? Or is this just a de­scrip­tion of cur­rent events be­ing a gen­eral grad­ual col­lapse?

This seems like what is hap­pen­ing now, and has been for a while. Ex­ist­ing ML sys­tems are clearly mak­ing Type-I prob­lems, already quite bad be­fore ML was a thing at all, much worse, to the ex­tent that I don’t see much abil­ity left of our civ­i­liza­tion to get any­thing that can’t be mea­sured in a short term feed­back loop—even in spaces like this, ap­peals to non-mea­surable or non-ex­plicit con­cerns are a near-im­pos­si­ble sell.

Part II prob­lems are not yet com­ing from ML sys­tems, ex­actly, But we cer­tainly have al­gorithms that are effec­tively op­ti­mized and se­lected for the abil­ity to gain in­fluence; the al­gorithm gains in­fluence, which causes peo­ple to care about it and feed into it, caus­ing it to get more. If we get less di­rect in the metaphor we get the same thing with memet­ics, cul­ture, life strate­gies, cor­po­ra­tions, me­dia prop­er­ties and so on. The em­pha­sis on choos­ing win­ners, be­ing ‘on the right side of his­tory’, sup­port­ing those who are good at get­ting sup­port. OP notes that this hap­pens in non-ML situ­a­tions ex­plic­itly, and there’s no clear di­vid­ing line in any case.

So if there is an­other the­ory that says, this has already hap­pened, what would one do next?

[John_Maxwell]

You could always get a job at a com­pany which con­trols an im­por­tant al­gorithm.

[ricraz]

Even­tu­ally we reach the point where we could not re­cover from a cor­re­lated au­toma­tion failure. Un­der these con­di­tions in­fluence-seek­ing sys­tems stop be­hav­ing in the in­tended way, since their in­cen­tives have changed—they are now more in­ter­ested in con­trol­ling in­fluence af­ter the re­sult­ing catas­tro­phe then con­tin­u­ing to play nice with ex­ist­ing in­sti­tu­tions and in­cen­tives.

I’m not sure I un­der­stand this part. The in­fluence-seek­ing sys­tems which have the most in­fluence also have the most to lose from a catas­tro­phe. So they’ll be in­cen­tivised to po­lice each other and make catas­tro­phe-avoidance mechanisms more ro­bust.

As an anal­ogy: we may already be past the point where we could re­cover from a cor­re­lated “world leader failure”: ev­ery world leader si­mul­ta­neously launch­ing a coup. But this doesn’t make such a failure very likely, un­less world lead­ers also have strong co­or­di­na­tion and com­mit­ment mechanisms be­tween them­selves (which are bind­ing even af­ter the catas­tro­phe).

[Wei_Dai]

(Upvoted be­cause I think this de­serves more clar­ifi­ca­tion/​dis­cus­sion.)

I’m not sure I un­der­stand this part. The in­fluence-seek­ing sys­tems which have the most in­fluence also have the most to lose from a catas­tro­phe. So they’ll be in­cen­tivised to po­lice each other and make catas­tro­phe-avoidance mechanisms more ro­bust.

I’m not sure ei­ther, but I think the idea is that once in­fluence-seek­ing sys­tems gain a cer­tain amount of in­fluence, it may be­come faster or more cer­tain for them to gain more in­fluence by caus­ing a catas­tro­phe than to con­tinue to work within ex­ist­ing rules and in­sti­tu­tions. For ex­am­ple they may pre­dict that un­less they do that, hu­mans will even­tu­ally co­or­di­nate to take back the in­fluence that hu­mans lost, or they may pre­dict that dur­ing such a catas­tro­phe they can prob­a­bly ex­pro­pri­ate a lot of re­sources cur­rently owned by hu­mans and gain much in­fluence that way, or hu­mans will vol­un­tar­ily hand more power to them in or­der to try to use them to deal with the catas­tro­phe.

As an anal­ogy: we may already be past the point where we could re­cover from a cor­re­lated “world leader failure”: ev­ery world leader si­mul­ta­neously launch­ing a coup. But this doesn’t make such a failure very likely, un­less world lead­ers also have strong co­or­di­na­tion and com­mit­ment mechanisms be­tween them­selves (which are bind­ing even af­ter the catas­tro­phe).

I think such a failure can hap­pen with­out es­pe­cially strong co­or­di­na­tion and com­mit­ment mechanisms. Some­thing like this hap­pened dur­ing the Chi­nese War­lord Era, when many mil­i­tary com­man­ders be­came war­lords dur­ing a cor­re­lated “mil­i­tary com­man­der failure”, and similar things prob­a­bly hap­pened many times through­out his­tory. I think what’s ac­tu­ally pre­vent­ing a “world leader failure” to­day is that most world lead­ers, es­pe­cially of the rich demo­cratic coun­tries, don’t see any way to fur­ther their own val­ues by launch­ing coups in a cor­re­lated way. In other words, what would they do af­ter­wards if they did launch such a coup, that would be bet­ter than just ex­er­cis­ing the power that they already have?

[ricraz]

I think the idea is that once in­fluence-seek­ing sys­tems gain a cer­tain amount of in­fluence, it may be­come faster or more cer­tain for them to gain more in­fluence by caus­ing a catas­tro­phe than to con­tinue to work within ex­ist­ing rules and in­sti­tu­tions.

The key is­sue here is whether there will be co­or­di­na­tion be­tween a set of in­fluence-seek­ing sys­tems that can cause (and will benefit from) a catas­tro­phe, even when other sys­tems are op­pos­ing them. If we pic­ture sys­tems as hav­ing power com­pa­rable to what com­pa­nies have now, that seems difficult. If we pic­ture them as hav­ing power com­pa­rable to what coun­tries have now, that seems fairly easy.

[Wei_Dai]

The key is­sue here is whether there will be co­or­di­na­tion be­tween a set of in­fluence-seek­ing sys­tems that can cause (and will benefit from) a catas­tro­phe, even when other sys­tems are op­pos­ing them.

Do you not ex­pect this thresh­old to be crossed sooner or later, as­sum­ing AI al­ign­ment re­mains un­solved? Also, it seems like the main al­ter­na­tive to this sce­nario is that the in­fluence-seek­ing sys­tems ex­pect to even­tu­ally gain con­trol of most of the uni­verse any­way (even with­out a “cor­re­lated au­toma­tion failure”), so they don’t see a rea­son to “rock the boat” and try to dis­pos­sess hu­mans of their re­main­ing in­fluence/​power/​re­sources, but this is al­most as bad as the “cor­re­lated au­toma­tion failure” sce­nario from an as­tro­nom­i­cal waste per­spec­tive. (I’m won­der­ing if you’re ques­tion­ing whether things will turn out badly, or ques­tion­ing whether things will turn out badly this way.)

[ricraz]

Mostly I am ques­tion­ing whether things will turn out badly this way.

Do you not ex­pect this thresh­old to be crossed sooner or later, as­sum­ing AI al­ign­ment re­mains un­solved?

Prob­a­bly, but I’m pretty un­cer­tain about this. It de­pends on a lot of messy de­tails about re­al­ity, things like: how offense-defence bal­ance scales; what pro­por­tion of pow­er­ful sys­tems are mostly al­igned; whether in­fluence-seek­ing sys­tems are risk-neu­tral; what self-gov­er­nance struc­tures they’ll set up; the ex­tent to which their prefer­ences are com­pat­i­ble with ours; how hu­man-com­pre­hen­si­ble the most im­por­tant up­com­ing sci­en­tific ad­vances are.

[Tobias_Baumann]

I agree with you that the “stereo­typed image of AI catas­tro­phe” is not what failure will most likely look like, and it’s great to see more dis­cus­sion of al­ter­na­tive sce­nar­ios. But why ex­actly should we ex­pect that the prob­lems you de­scribe will be ex­ac­er­bated in a fu­ture with pow­er­ful AI, com­pared to the state of con­tem­po­rary hu­man so­cieties? Hu­mans also of­ten op­ti­mise for what’s easy to mea­sure, es­pe­cially in or­gani­sa­tions. Is the con­cern that cur­rent ML sys­tems are un­able to op­ti­mise hard-to-mea­sure goals, or goals that are hard to rep­re­sent in a com­put­er­ised form? That is true but I think of this as a limi­ta­tion of con­tem­po­rary ML ap­proaches rather than a fun­da­men­tal prop­erty of ad­vanced AI. With gen­eral in­tel­li­gence, it should also be pos­si­ble to op­ti­mise goals that are hard-to-mea­sure.

Similarly, hu­mans /​ com­pa­nies /​ or­gani­sa­tions reg­u­larly ex­hibit in­fluence-seek­ing be­havi­our, and this can cause harm but it’s also usu­ally pos­si­ble to keep it in check to at least a cer­tain de­gree.

So, while you point at things that can plau­si­bly go wrong, I’d say that these are peren­nial is­sues that may be­come bet­ter or worse dur­ing and af­ter the tran­si­tion to ad­vanced AI, and it’s hard to pre­dict what will hap­pen. Of course, this does not make a very ap­peal­ing tale of doom – but maybe it would be best to dis­pense with tales of doom al­to­gether.

I’m also not yet con­vinced that “these cap­ture the most im­por­tant dy­nam­ics of catas­tro­phe.” Speci­fi­cally, I think the fol­low­ing are also po­ten­tially se­ri­ous is­sues:
- Un­for­tu­nate cir­cum­stances in fu­ture co­op­er­a­tion prob­lems be­tween AI sys­tems (and /​ or hu­mans) re­sult in wide­spread defec­tion, lead­ing to poor out­comes for ev­ery­one.
- Con­flicts be­tween key fu­ture ac­tors (AI or hu­man) re­sult in large quan­tities of dis­value (agen­tial s-risks).
- New tech­nol­ogy leads to rad­i­cal value drift of a form that we wouldn’t en­dorse.

[paulfchristiano]

But why ex­actly should we ex­pect that the prob­lems you de­scribe will be ex­ac­er­bated in a fu­ture with pow­er­ful AI, com­pared to the state of con­tem­po­rary hu­man so­cieties?

To a large ex­tent “ML” refers to a few par­tic­u­lar tech­nolo­gies that have the form “try a bunch of things and do more of what works” or “con­sider a bunch of things and then do the one that is pre­dicted to work.”

That is true but I think of this as a limi­ta­tion of con­tem­po­rary ML ap­proaches rather than a fun­da­men­tal prop­erty of ad­vanced AI.

I’m mostly aiming to de­scribe what I think is in fact most likely to go wrong, I agree it’s not a gen­eral or nec­es­sary fea­ture of AI that its com­par­a­tive ad­van­tage is op­ti­miz­ing easy-to-mea­sure goals.

(I do think there is some real sense in which get­ting over this re­quires “solv­ing al­ign­ment.”)

[John_Maxwell]

To a large ex­tent “ML” refers to a few par­tic­u­lar tech­nolo­gies that have the form “try a bunch of things and do more of what works” or “con­sider a bunch of things and then do the one that is pre­dicted to work.”

Why not “try a bunch of mea­sure­ments and figure out which one gen­er­al­izes best” or “con­sider a bunch of things and then do the one that is pre­dicted to work ac­cord­ing to the broad­est va­ri­ety of ML-gen­er­ated mea­sure­ments”? (I ex­pect there’s already some re­search cor­re­spond­ing to these sug­ges­tions, but more could be valuable?)

[Jan_Kulveit]

Rea­sons for some care­ful optimism

in Part I., it can be the case that hu­man val­ues are ac­tu­ally com­plex com­bi­na­tion of easy to mea­sure goals + com­plex world mod­els, so the struc­ture of the prox­ies will be able to rep­re­sent what we re­ally care about. (I don’t know. Also the re­sult can still stop rep­re­sent our val­ues with fur­ther scal­ing and evolu­tion.)

in Part II., it can be the case that in­fluence-seek­ing pat­terns are more com­pu­ta­tion­ally costly than straight­for­ward pat­terns, and they can be in part sup­pressed by op­ti­mis­ing for pro­cess­ing costs, bounded-ra­tio­nal­ity style. To some ex­tend, in­fluence-seek­ing pat­terns at­tempt­ing to grow and con­trol the whole sys­tem seems to me to be some­thing hap­pen­ing also within our own minds. I would guess some com­bi­na­tional of im­mune sys­tem + metacog­ni­tion + bounded ra­tio­nal­ity + sta­bil­i­sa­tion by com­plex­ity is sta­bil­is­ing many hu­man minds. (I don’t know if any­thing of that can scale ar­bi­trar­ily.)

[John_Maxwell]

Once we start search­ing over poli­cies that un­der­stand the world well enough, we run into a prob­lem: any in­fluence-seek­ing poli­cies we stum­ble across would also score well ac­cord­ing to our train­ing ob­jec­tive, be­cause perform­ing well on the train­ing ob­jec­tive is a good strat­egy for ob­tain­ing in­fluence.

...

One rea­son to be scared is that a wide va­ri­ety of goals could lead to in­fluence-seek­ing be­hav­ior, while the “in­tended” goal of a sys­tem is a nar­rower tar­get, so we might ex­pect in­fluence-seek­ing be­hav­ior to be more com­mon in the broader land­scape of “pos­si­ble cog­ni­tive poli­cies.”

Con­sider this video of an AI sys­tem with a mis­speci­fied re­ward func­tion. (Back­ground in this post.) The AI sys­tem searches the space of poli­cies to dis­cover the one that performs best ac­cord­ing to its re­ward func­tion in the simu­lated boat-rac­ing world. It turns out that the one which performs best ac­cord­ing to this mis­speci­fied re­ward func­tion doesn’t perform well ac­cord­ing to the in­tended re­ward func­tion (the “train­ing ob­jec­tive” that the sys­tem’s de­vel­op­ers use to eval­u­ate perfor­mance).

The goal of pick­ing up as many power-ups as pos­si­ble could lead to in­fluence-seek­ing be­hav­ior: If the boat can per­suade us to leave the simu­la­tion on, it can keep pick­ing up power-ups un­til the end of time. Sup­pose for the sake of ar­gu­ment that perform­ing well on the train­ing ob­jec­tive is the best strat­egy for ob­tain­ing in­fluence, as you posit. Then the boat should com­plete the race cor­rectly, in or­der to fool us into think­ing it re­li­ably works to­wards the train­ing ob­jec­tive. And yet it doesn’t com­plete the race cor­rectly in the video. Why not?

One an­swer is that the hu­man su­per­vi­sor isn’t part of the sys­tem’s world model. But I don’t think that would change things. Sup­pose in­stead of mak­ing use of an ex­ist­ing video game, the sys­tem’s world model was gen­er­ated au­to­mat­i­cally by ob­serv­ing the world, and the ob­ser­va­tions were de­tailed enough to in­clude the su­per­vi­sor of the AI sys­tem and even the AI sys­tem it­self. Now the boat is try­ing to find poli­cies that max­i­mize power-ups in this ab­surdly de­tailed, au­to­mat­i­cally gen­er­ated world model (with some power-ups man­u­ally added in). Why would a policy which ma­nipu­lates the op­er­a­tor within the simu­lated world score well? It seems like it would take a con­fused world model for ma­nipu­la­tion of the simu­lated op­er­a­tor to help with pick­ing up simu­lated power-ups. Like if paint­ing an­i­mals on cave walls ac­tu­ally caused them to ap­pear. Larry Elli­son is not go­ing to win a yacht race by tel­ling his data sci­en­tist to crip­ple his op­po­nents in a simu­la­tion.

[Another frame: Carte­sian du­al­ism will hap­pen by de­fault, or at least will be easy to en­force on the ar­chi­tec­tural level. You could ar­gue Carte­sian du­al­ists lose be­cause they don’t do self-im­prove­ment? But an im­plied premise of your post is that foom won’t hap­pen. I dis­agree but that’s an­other dis­cus­sion.]

But let’s sup­pose the world model ac­tu­ally is con­fused, and the best policy in the simu­la­tion is one that ma­nipu­lates the simu­lated op­er­a­tor to gain simu­lated power-ups. Even in this case, I think we’d still see a video like I linked ear­lier. We’d see the boat pow­er­ing over to the part of the simu­lated world where the simu­lated op­er­a­tor re­sides, do­ing some­thing to ma­nipu­late the simu­lated op­er­a­tor, and then the boat would have loads of power-ups some­how. I think the biggest con­cern is ex­po­sure to an in­for­ma­tion haz­ard when we see how the boat ma­nipu­lates the op­er­a­tor. (Luck­ily, if we im­ple­ment an in­for­ma­tion haz­ard filter be­fore let­ting our­selves watch the video, the boat will not op­ti­mize to get past it.)

Hu­man billion­aires are hiring physi­cists to try & figure out if our uni­verse is a simu­la­tion and if so, how to hack our way out. So there might be some­thing here. Maybe if world model con­struc­tion hap­pens in tan­dem with ex­plor­ing the space of poli­cies, the boat will start “con­sid­er­ing the pos­si­bil­ity that it’s in a simu­la­tion” in a sense. (Will try­ing to ma­nipu­late the thing con­trol­ling the simu­la­tion be a policy that performs well in the simu­la­tion?)

[paulfchristiano]

I’m not mostly wor­ried about in­fluence-seek­ing be­hav­ior emerg­ing by “spec­ify a goal” --> “get­ting in­fluence is the best way to achieve that goal.” I’m mostly wor­ried about in­fluence-seek­ing be­hav­ior emerg­ing within a sys­tem by virtue of se­lec­tion within that pro­cess (and by ran­dom­ness at the low­est level).

[John_Maxwell]

OK, thanks for clar­ify­ing. Sounds like a new fram­ing of the “dae­mon” idea.

[Wei_Dai]

Sounds like a new fram­ing of the “dae­mon” idea.

That’s my im­pres­sion as well. If it’s cor­rect, seems like it would be a good idea to men­tion that ex­plic­itly in the post, so peo­ple can link up the new con­cept with their old con­cept.

[TurnTrout]

So the con­cern here is that even if the goal, say, ro­bustly pe­nal­izes gain­ing in­fluence, the agent still has in­ter­nal se­lec­tion pres­sures for seek­ing in­fluence? And this might not be pe­nal­ized by the outer crite­rion if the policy plays nice on-dis­tri­bu­tion?

[vlad_m]

The goal that the agent is se­lected to score well on is not nec­es­sar­ily the goal that the agent is it­self pur­su­ing. So, un­less the agent’s in­ter­nal goal matches the goal for which it’s se­lected, the agent might still seek in­fluence be­cause its in­ter­nal goal per­mits that. I think this is in part what Paul means by “Avoid­ing end-to-end op­ti­miza­tion may help pre­vent the emer­gence of in­fluence-seek­ing be­hav­iors (by im­prov­ing hu­man un­der­stand­ing of and hence con­trol over the kind of rea­son­ing that emerges)”

[TurnTrout]

And if the in­ter­nal goal doesn’t per­mit that? I’m try­ing to feel out which lev­els of meta are prob­le­matic in this situ­a­tion.

[habryka]

Pro­moted to cu­rated: I think this post made an im­por­tant ar­gu­ment, and did so in a way that I ex­pect the post and the re­sult­ing dis­cus­sion around it to func­tion as a refer­ence-work for quite a while.

In ad­di­tion to the post it­self, I also thought the dis­cus­sion around it was quite good and helped me clar­ify my think­ing in this do­main a good bit.

[Wei_Dai]

There’s a bunch of bul­let points be­low Part 1 and Part 2. Are these in­tended to be par­allel with them on the same level, or in­stances/​sub­cat­e­gories of them?

Oh, this is only on GW. On LW it looks very differ­ent. Pre­sum­ably the LW ver­sion is the in­tended ver­sion.

[clone of saturn]

Oops, this bug should be fixed now.

[shminux]

Paul, is any­one at MIRI or el­se­where do­ing nu­mer­i­cal simu­la­tion of your ideas? Or are those just open-loop thoughts?

[ESRogs]

Not sure ex­actly what you mean by “nu­mer­i­cal simu­la­tion”, but you may be in­ter­ested in https://​​ought.org/​​ (where Paul is a col­lab­o­ra­tor), or in Paul’s work at OpenAI: https://​​ope­nai.com/​​blog/​​au­thors/​​paul/​​ .

[Virgil Kurkjian]

Po­ten­tially rele­vant is my post Leto among the Machines, where I dis­cuss the early stages of what you aptly call “get what you can mea­sure”.

[Donald Hobson]

As far as I un­der­stand it, you are propos­ing that the most re­al­is­tic failure mode con­sists of many AI sys­tems, all put into a po­si­tion of power by hu­mans, and op­ti­miz­ing for their own prox­ies. Call these Trusted Trial and Er­ror AI’s (TTE)

The dis­t­in­guish­ing fea­tures of TTE’s are that they were Trusted. A hu­man put them in a po­si­tion of power. Hu­mans have re­fined, un­der­stood and checked the code enough that they are pre­pared to put this al­gorithm in a self driv­ing car, or a stock man­age­ment sys­tem. They are not lab pro­to­types. They are also Trial and er­ror learn­ers, not one shot learn­ers.

Some More de­scrip­tions of what ca­pa­bil­ity range I am con­sid­er­ing.

Sup­pose hy­po­thet­i­cally that we had TTE re­in­force­ment learn­ers, a lit­tle bet­ter than to­days state of the art, and noth­ing be­yond that. The AI’s are ad­vanced enough that they can take a moun­tain of med­i­cal data and train them­selves to be skil­led doc­tors by trial and er­ror. How­ever they are not ad­vanced enough to figure out how hu­mans work from, say a se­quenced genome and noth­ing more.

Give them con­trol of all the traf­fic lights in a city, and they will learn how to min­i­mize traf­fic jams. They will ar­range for peo­ple to drive in cir­cles rather than stay still, so that they do not count as part of a traf­fic jam. How­ever they will not do any­thing out­side their pre­set policy space, like hack­ing into the traf­fic light con­trol sys­tem of other cities, or de­stroy­ing the city with nukes.

If such tech­nol­ogy is eas­ily available, peo­ple will start to use it for things. Some peo­ple put it in po­si­tions of power, oth­ers are more hes­i­tant. As the only way the sys­tem can learn to avoid some­thing is through trial and er­ror, the sys­tem has to cause a (prob­a­bly sev­eral) pub­lic out­crys be­fore it learns not to do so. If no one told the traf­fic light sys­tem that car crashes are bad on simu­la­tions or past data, (Align­ment failure) Then even if pub­lic opinion feeds di­rectly into re­ward, it will have to cause sev­eral car crashes that are clearly its fault be­fore it learns to only cause crashes that can be blamed on some­one else. How­ever, de­liber­ately caus­ing crashes will prob­a­bly get the sys­tem shut off or se­ri­ously mod­ified.

Note that we are sup­pos­ing many of these sys­tems ex­ist­ing, so the failures of some, com­bined with plenty of simu­lated failures, will give us a good idea of the failure modes.

The space of bad things an AI can get away with is small and highly com­plex in the space of bad things. An TTE set to re­duce crime rates tries mak­ing the crime re­port forms longer, this re­duces re­ported crime, but hu­mans quickly re­al­ize what its do­ing. It would have to do this and be patched many times be­fore it came up with a method that hu­mans wouldn’t no­tice.

Given Ad­vanced TTE’s as the most ad­vanced form of AI, we might slowly de­velop a prob­lem, but the de­ploy­ment of TTE’s would be slowed by the time it takes to gather data and check re­li­a­bil­ity. Espe­cially given mis­trust af­ter sev­eral ma­jor failures. And I sus­pect that due to statis­ti­cal similar­ity of train­ing and test­ing, many differ­ent sys­tems op­ti­miz­ing differ­ent prox­ies, and hu­mans hav­ing the best ab­stract rea­son­ing about novel situ­a­tions, and the power to turn the sys­tems off, any dis­crep­ancy of goals will be mod­er­ately minor. I do not ex­pect such op­ti­miza­tion power to be sig­nifi­cantly more pow­er­ful or less al­igned than mod­ern cap­i­tal­ism.

This all as­sumes that no one will man­age to make a lin­ear time AIXI. If such a thing is made, it will break out of any boxes and take over the world. So, we have a so­cial pro­cess of adap­tion to TTE AI, which is already in its early stages with things like self driv­ing cars, and at any time, this pro­cess could be ren­dered ir­rele­vant by the ar­rival of a su­per-in­tel­li­gence.

[avturchin]

Just be­fore read­ing this, I got a shower thought that most AI-re­lated catas­tro­phes de­scribed pre­vi­ously were of “hy­per-ra­tio­nal” type, e.g. the pa­per­clip­per, which from first prin­ci­ples de­cides that it must pro­duce in­finitely many pa­per­clips.

How­ever, this is not how ML-based sys­tems fail. They ei­ther fall ran­domly, when en­counter some­thing like ad­ver­sar­ial ex­am­ple, or fall slowly, by good­heart­ing some perfor­mance mea­sure. Such sys­tems could be also used to cre­ate dan­ger­ous weapons, e.g. fak­e­news or viruses, or in­ter­act un­pre­dictably with each other.

Fu­ture GPT-3 will be pro­tected from hy­per-ra­tio­nal failures be­cause of the noisy na­ture of its an­swers, so it can’t stick for­ever to some wrong policy.

[Daniel Kokotajlo]

I think that’s a straw man of the clas­sic AI-re­lated catas­tro­phe sce­nar­ios. Bostrom’s “covert prepa­ra­tion” --> “Treach­er­ous turn” --> “takeover” story maps pretty nicely to Paul’s “seek in­fluence via gam­ing tests” --> “they are now more in­ter­ested in con­trol­ling in­fluence af­ter the re­sult­ing catas­tro­phe then con­tin­u­ing to play nice with ex­ist­ing in­sti­tu­tions and in­cen­tives” --> ” One day lead­ers may find that de­spite their nom­i­nal au­thor­ity they don’t ac­tu­ally have con­trol over what these in­sti­tu­tions do. For ex­am­ple, mil­i­tary lead­ers might is­sue an or­der and find it is ig­nored. This might im­me­di­ately prompt panic and a strong re­sponse, but the re­sponse it­self may run into the same prob­lem, and at that point the game may be up. ”

[TheWakalix]

the pa­per­clip­per, which from first prin­ci­ples de­cides that it must pro­duce in­finitely many paperclips

I don’t think this is an ac­cu­rate de­scrip­tion of the pa­per­clip sce­nario, un­less “first prin­ci­ples” means “hard­coded goals”.

Fu­ture GPT-3 will be pro­tected from hy­per-ra­tio­nal failures be­cause of the noisy na­ture of its an­swers, so it can’t stick for­ever to some wrong policy.

Ig­nor­ing how GPT isn’t agen­tic and hand­wav­ing an agen­tic analogue, I don’t think this is sound. Wrong poli­cies make up al­most all of poli­cys­pace; the prob­lem is not that the AI might en­ter a spe­cial state of wrong­ness, it’s that the AI might leave the spe­cial state of cor­rect­ness. And to the ex­tent that GPT is hin­dered by its ran­dom­ness, it’s un­able to carry out long-term plans at all—it’s safe only be­cause it’s weak.