Davidmanheim

• Davidmanheim 4 Oct 2019 6:53 UTC

  LW: 16 AF: 5

  AF

  on: De­bate on In­stru­men­tal Con­ver­gence be­tween LeCun, Rus­sell, Ben­gio, Zador, and More

  I com­mented on the thread (af­ter see­ing this) in or­der to add a link to my pa­per that ad­dresses Ben­gio’s last ar­gu­ment;

  @YoshuaBen­gio I at­tempted to for­mal­ize this ar­gu­ment some­what in a re­cent pa­per. I don’t think the ar­gu­ment there is par­tic­u­larly air­tight, but I think it pro­vides a sig­nifi­cantly stronger ar­gu­ment for why we should be­lieve that in­ter­ac­tion be­tween op­ti­miz­ing sys­tems is fun­da­men­tally hard.

  https://​www.mdpi.com/​2504-2289/​3/​2/​21/​htm

  Paper ab­stract: “An im­por­tant challenge for safety in ma­chine learn­ing and ar­tifi­cial in­tel­li­gence sys­tems is a set of re­lated failures in­volv­ing speci­fi­ca­tion gam­ing, re­ward hack­ing, frag­ility to dis­tri­bu­tional shifts, and Good­hart’s or Camp­bell’s law. This pa­per pre­sents ad­di­tional failure modes for in­ter­ac­tions within multi-agent sys­tems that are closely re­lated. Th­ese multi-agent failure modes are more com­plex, more prob­le­matic, and less well un­der­stood than the sin­gle-agent case, and are also already oc­cur­ring, largely un­no­ticed. After mo­ti­vat­ing the dis­cus­sion with ex­am­ples from poker-play­ing ar­tifi­cial in­tel­li­gence (AI), the pa­per ex­plains why these failure modes are in some senses un­avoid­able. Fol­low­ing this, the pa­per cat­e­go­rizes failure modes, pro­vides defi­ni­tions, and cites ex­am­ples for each of the modes: ac­ci­den­tal steer­ing, co­or­di­na­tion failures, ad­ver­sar­ial mis­al­ign­ment, in­put spoofing and fil­ter­ing, and goal co-op­tion or di­rect hack­ing. The pa­per then dis­cusses how ex­tant liter­a­ture on multi-agent AI fails to ad­dress these failure modes, and iden­ti­fies work which may be use­ful for the miti­ga­tion of these failure modes.”

• Davidmanheim 2 Oct 2019 10:42 UTC

  4 points

  in reply to: Wei_Dai's comment on: Fol­low-Up to Petrov Day, 2019

  Re: ex­am­ples of point #1, I don’t think that sham­ing in this fo­rum is pro­duc­tive—it’s po­lariz­ing and stig­ma­tiz­ing rather than helpful. But I do know of sev­eral in­di­vi­d­u­als and a cou­ple or­ga­ni­za­tions which are guilty of this, each re­peat­edly.

  I do think that peo­ple should be more will­ing to per­son­ally /​ pri­vately re­spond if some­one does some­thing, and I have done so in sev­eral spe­cific cases where some­one de­cided on a unilat­er­al­ist ap­proach that I thought was dam­ag­ing.

• Davidmanheim 29 Sep 2019 10:08 UTC

  23 points

  in reply to: Wei_Dai's comment on: Fol­low-Up to Petrov Day, 2019

  Even ig­nor­ing the above prob­lem, I’m con­fused why it’s valuable to build up a “real tra­di­tion” among LW users, given that the wider unilat­er­al­ist curse prob­lem that our world faces can’t pos­si­bly be solved by LW users hav­ing such a tra­di­tion.

  A few points.

  First, I don’t think it’s clear that in the Ra­tion­al­ist /​ EA com­mu­nity, there is enough re­in­force­ment of this, and I rou­tinely see is­sues with peo­ple “go­ing rogue” and unilat­er­ally en­gag­ing in ac­tivi­ties that oth­ers have warned them would be dan­ger­ous, net nega­tive, etc.

  Se­cond, it’s valuable even as an ex­em­plar; we should be able to say that there is such a com­mu­nity, and that they are ca­pa­ble of ex­er­cis­ing at least this min­i­mal level of re­straint.

  Third, I think it’s clear that in the next decade the num­ber of peo­ple in the Ra­tion­al­ist-sphere that are in ac­tual po­si­tions of (rel­a­tively sig­nifi­cant) power will con­tinue to grow, and we have already seen some such peo­ple emerge in gov­ern­ment and in the world of NGOs. For AI, in par­tic­u­lar, there are many peo­ple who have sig­nifi­cant in­fluence in mak­ing de­ci­sions that could sig­nifi­cantly af­fect Hu­man­ity’s fu­ture. Their ac­tive (i.e. pas­sive) par­ti­ci­pa­tion in this seems likely to at least give them a bet­ter un­der­stand­ing of what is needed when they are faced with these choices.

• Davidmanheim 17 Sep 2019 20:28 UTC

  3 points

  in reply to: clone of saturn's comment on: Diver­gence on Ev­i­dence Due to Differ­ing Pri­ors—A Poli­ti­cal Case Study

  They don’t con­tra­dict di­rectly, but they re­flect nearly in­com­pat­i­ble up­dates to their world-mod­els based on the same data.

Diver­gence on Ev­i­dence Due to Differ­ing Pri­ors—A Poli­ti­cal Case Study

• Davidmanheim 12 Sep 2019 6:54 UTC

  LW: 2 AF: 1

  AF

  in reply to: Wei_Dai's comment on: Hack­able Re­wards as a Safety Valve?

  But I don’t un­der­stand how you can ex­pect this (i.e., non-SAI-con­cerned AI safety work that make easy-to-sub­vert chan­nels harder to hit) to not hap­pen, or to make it sig­nifi­cantly less likely to hap­pen, given that peo­ple want to build AIs that do things beside re­ward hack­ing

  I was mostly not­ing that I hadn’t thought of this, hadn’t seen it men­tioned, and so my model for re­turns to non-fun­da­men­tal al­ign­ment AI safety in­vest­ments didn’t pre­vi­ously ac­count for this. Reflect­ing on that fact now, I think the key strate­gic im­pli­ca­tion re­lates to the on­go­ing de­bate about pri­ori­ti­za­tion of effort in AI-safety.

  (Now, some low con­fi­dence spec­u­la­tion on this:) Peo­ple who be­lieve that near-term Foom! is rel­a­tively un­likely, but worry about mis­al­igned non-su­per­in­tel­li­gent NAI/​Near-hu­man AGI, may be mak­ing the Foom! sce­nario more likely. That means that at­ten­tion to AI safety that pushes for “safer self-driv­ing cars” and “re­duc­ing and miti­gat­ing side-effects” is plau­si­bly a net nega­tive if done poorly, in­stead of be­ing be­nign.

• Davidmanheim 11 Sep 2019 9:08 UTC

  LW: 2 AF: 1

  AF

  in reply to: Wei_Dai's comment on: Hack­able Re­wards as a Safety Valve?

  My claim here is that su­per­in­tel­li­gence is a re­sult of train­ing, not a start­ing con­di­tion. Yes, a SAI would do bad things un­less ro­bustly al­igned, but build­ing the SAI re­quires it not to wire­head at an ear­lier stage in the pro­cess. My claim is that I am un­sure that there is a way to train such a sys­tem that was not built with safety in mind such that it gets to a point where it is more likely to gain in­tel­li­gence than it is to find ways to re­ward hack—not nec­es­sar­ily via di­rect ac­cess, but via what­ever chan­nel is cheap­est. And mak­ing easy-to-sub­vert chan­nels harder to hit seems to be the fo­cus of a fair amount of non-SAI-con­cerned AI safety work, which seems like a net-nega­tive.

Hack­able Re­wards as a Safety Valve?

• Davidmanheim 5 Sep 2019 11:19 UTC

  3 points

  in reply to: redlizard's comment on: What Pro­gram­ming Lan­guage Char­ac­ter­is­tics Would Allow Prov­ably Safe AI?

  This is re­ally helpful—thanks!

• Davidmanheim 29 Aug 2019 9:05 UTC

  6 points

  in reply to: Donald Hobson's comment on: What Pro­gram­ming Lan­guage Char­ac­ter­is­tics Would Allow Prov­ably Safe AI?

  Yes, an un­safe AI can­not be boxed on un­safe hard­ware, not can any AI run­ning on phys­i­cal hard­ware be made im­mune to at­tacks—but those are very differ­ent ques­tions. For this ques­tion, first we as­sume that a prov­ably safe AI can be writ­ten, then I wanted to ask what lan­guage would be needed.

[Question] What Pro­gram­ming Lan­guage Char­ac­ter­is­tics Would Allow Prov­ably Safe AI?

• Davidmanheim 26 Aug 2019 10:13 UTC

  2 points

  in reply to: Gurkenglas's comment on: Where are peo­ple think­ing and talk­ing about global co­or­di­na­tion for AI safety?

  No, that’s im­plicit in the model—and ei­ther *some* crisis re­quiring higher ca­pac­ity than we have will over­whelm us and we’ll all die (and it doesn’t mat­ter which,) or the var­i­ance is rel­a­tively small so no such event oc­curs, and/​or our ca­pac­ity to man­age risks grows quickly enough that we avoid the up­per tail.

• Davidmanheim 20 Aug 2019 11:18 UTC

  2 points

  in reply to: Matthew Barnett's comment on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  Fully agree—I was us­ing the ex­am­ple to make a far less fun­da­men­tal point.

• Davidmanheim 19 Aug 2019 9:56 UTC

  LW: 2 AF: 1

  AF

  in reply to: Pattern's comment on: Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

  2) Cal­ling the is­sues be­tween the agents be­cause of model differ­ences “ter­minol­ogy is­sues” could also work well—this may be a lit­tle like peo­ple talk­ing past each other.

  I re­ally like this point. I think it’s par­allel to the hu­man is­sue where differ­ent mod­els of the world can lead to mis­in­ter­pre­ta­tion of the “same” goal. So “ter­minol­ogy is­sues” would in­clude, for ex­am­ple, two differ­ent mea­sure­ments of what we would as­sume is the same quan­tity. If the base op­ti­mizer is look­ing to set the tem­per­a­ture and us­ing a wall-ther­mome­ter, while the mesa-op­ti­mizer is us­ing one lo­cated on the floor, the mesa-op­ti­mizer might be mis-al­igned be­cause it in­ter­prets “tem­per­a­ture” as refer­ring to a differ­ent fact than the base-op­ti­mizer. On the other hand, when the same met­ric is be­ing used by both par­ties, the class of pos­si­ble mis­takes does not in­clude what we’re not call­ing ter­minol­ogy is­sues.

  I think this also points to a fun­da­men­tal episte­molog­i­cal is­sue, one even broader than goal-rep­re­sen­ta­tion. It’s pos­si­ble that two mod­els dis­agree on rep­re­sen­ta­tion, but agree on all ob­ject level claims—think of us­ing differ­ent co­or­di­nate sys­tems. Be­cause ter­minol­ogy is­sues can cause mis­takes, I’d sug­gest that agents with non-shared world mod­els can only re­li­ably com­mu­ni­cate via ob­ject-level claims.

  The im­pli­ca­tion for AI al­ign­ment might be that we need AI to ei­ther fun­da­men­tally model the world the same way as hu­mans, or need to com­mu­ni­cate only via ob­ject-level goals and con­straints.

• Davidmanheim 19 Aug 2019 9:44 UTC

  2 points

  in reply to: Pattern's comment on: Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

  It seems like there’s a differ­ence be­tween the two cases. If I write a pro­gram to take the CRT, and then we both take it, and we both get the same score (and that isn’t a perfect score), be­cause it solved them the way I solve them, that doesn’t sound like mis­al­ign­ment.

  The mis­al­ign­ment here is be­tween you and the CRT, and re­flects your model be­ing mis­al­igned with the goal /​ re­al­ity. That’s why I’m call­ing it a prin­ci­pal al­ign­ment failure—even though it’s the pro­gram /​ mesa-op­ti­mizer that fails, the al­ign­ment failure is lo­cated in the prin­ci­pal, you /​ the base op­ti­mizer.

• Davidmanheim 19 Aug 2019 9:22 UTC

  4 points

  on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  Separate from my other com­ment, I want to ques­tion your as­sump­tion that we must worry about an AI-take­off that is ex­po­nen­tially bet­ter than hu­mans at ev­ery­thing, so that a very slight mis­al­ign­ment would be dis­as­trous. That seems pos­si­ble, per Eliezer’s Rocket Ex­am­ple, but is far from cer­tain.

  It seems likely that in­stead there are fun­da­men­tal limits on in­tel­li­gence (for a given ar­chi­tec­ture, at least) and while it is un­likely that the over­all limits are co­in­ci­den­tally the same as /​ near hu­man in­tel­li­gence, it seems plau­si­ble that the first su­per­hu­man AI sys­tem still plateaus some­where far short of in­finite op­ti­miza­tion power. If so, we only need to miti­gate well, in­stead of perfectly al­ign the AI to our goals.

• Davidmanheim 19 Aug 2019 9:15 UTC

  10 points

  on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  I’ll start by not­ing that I am in the strange (for me) po­si­tion of ar­gu­ing that some­one is too con­cerned about over-op­ti­miza­tion failures, rather than try­ing to con­vince some­one who is dis­mis­sive. But that said, I do think that the con­cern here, while real, is miti­gable in a va­ri­ety of ways.

  First, there is the pos­si­bil­ity of re­duc­ing op­ti­miza­tion pres­sure. One key con­tri­bu­tion here is Jes­sica Tay­lor’s Quan­tiliz­ers pa­per, which you note, that shows a way to build sys­tems that op­ti­mize but are not nearly as sub­ject to Good­hart’s curse. I think you are too dis­mis­sive. Similarly, you are dis­mis­sive of op­ti­miz­ing the tar­get di­rectly. I think that the episte­molog­i­cal is­sues you point to are pos­si­ble to miti­gate to the ex­tent that they won’t cause mis­al­ign­ment be­tween re­al­ity and an AI’s rep­re­sen­ta­tion of that re­al­ity. Once that is done, the re­main­ing is­sue is al­ign­ing “true” goals with the mea­sured goals, which is still hard, but cer­tainly not fun­da­men­tally im­pos­si­ble in the same way.

  Se­cond, you note that you don’t think we will solve al­ign­ment. I agree, be­cause I think that “al­ign­ment” pre­sup­poses a sin­gle co­her­ent ideal. If hu­man prefer­ences are di­verse, as it seems they are, we may find that al­ign­ment is im­pos­si­ble. This, how­ever, al­lows a very differ­ent ap­proach. This would op­ti­mize only when it finds Pareto-im­prove­ments across a set of sub-al­ign­ment met­rics or goals, to con­strain the pos­si­bil­ity of run­away op­ti­miza­tion. Even if al­ign­ment is pos­si­ble, it seems likely that we can spec­ify a set of di­verse goals /​ met­rics that are all al­igned with some hu­man goals, so that the sys­tem will be limited in its abil­ity to be mis­al­igned.

  Lastly, there is op­ti­miza­tion for a safe and very limited goal. If the goal is limited and spe­cific, and we find a way to min­i­mize side-effects, this seems like it could be fairly safe. For ex­am­ple, Or­a­cle AIs are an at­tempt to severely limit the goal. More broadly, how­ever, we might be able to build con­straints that work, so that it that can re­li­ably perform limited tasks (“put a straw­berry on a plate with­out pro­duc­ing any catas­trophic side-effects.”)

Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

• Davidmanheim 12 Aug 2019 1:34 UTC

  LW: 2 AF: 1

  AF

  on: In­terthe­o­retic util­ity com­par­i­son: examples

  This is very in­ter­est­ing—I hadn’t thought about util­ity ag­gre­ga­tion for a sin­gle agent be­fore, but it seems clearly im­por­tant now that it has been pointed out.

  I’m think­ing about this in the con­text of both the hu­man brain as an amalga­ma­tion of sub-agents, and or­ga­ni­za­tions as an amalga­ma­tion of in­di­vi­d­u­als. Note that we can treat or­ga­ni­za­tions as ra­tio­nally max­i­miz­ing some util­ity func­tion in the same way we can treat in­di­vi­d­u­als as do­ing so—but I think that for many or most vot­ing or de­ci­sion struc­tures, we should be able to rule out the claim that they are fol­low­ing any weighted com­bi­na­tion of nor­mal­ized util­ities of the agents in­volved in the sys­tem us­ing any in­terthe­o­retic com­par­i­son. This seems like a use­ful re­sult if we can prove it. (Alter­na­tively, it may be that cer­tain de­ci­sion rules map to spe­cific in­terthe­o­retic com­par­i­son rules, which would be even more in­ter­est­ing.)

• Davidmanheim 8 Aug 2019 15:04 UTC

  2 points

  in reply to: jacobjacob's comment on: How can we re­spond to info-cas­cades? [Info-cas­cade se­ries]

  1) It’s nei­ther noise nor rapid in­crease—it’s de­layed feed­back. Con­trol the­o­rists in en­g­ineer­ing have this as a re­ally clear, ba­sic re­sult, that de­layed feed­back is re­ally re­ally bad in var­i­ous ways. There are en­tire books on how to do it well—https://​​books.google.ch/​​books?id=Cy_wCAAAQBAJ&pg=PR9&lpg=PR9 - but do­ing it with­out us­ing these more com­plex tech­niques is bad.

  2) You ei­ther hire a con­trol the­o­rist, or (more prac­ti­cally) you avoid the cur­rent feed­back mechanism, and in­stead get peo­ple on the phone to talk about and un­der­stand what ev­ery­one needs, as op­posed to rely­ing on their de­layed feed­back in the form of nu­meric or­ders.