Davidmanheim

• Davidmanheim 19 Aug 2019 9:56 UTC

  2 points

  AF

  in reply to: Pattern's comment on: Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

  2) Cal­ling the is­sues be­tween the agents be­cause of model differ­ences “ter­minol­ogy is­sues” could also work well—this may be a lit­tle like peo­ple talk­ing past each other.

  I re­ally like this point. I think it’s par­allel to the hu­man is­sue where differ­ent mod­els of the world can lead to mis­in­ter­pre­ta­tion of the “same” goal. So “ter­minol­ogy is­sues” would in­clude, for ex­am­ple, two differ­ent mea­sure­ments of what we would as­sume is the same quan­tity. If the base op­ti­mizer is look­ing to set the tem­per­a­ture and us­ing a wall-ther­mome­ter, while the mesa-op­ti­mizer is us­ing one lo­cated on the floor, the mesa-op­ti­mizer might be mis-al­igned be­cause it in­ter­prets “tem­per­a­ture” as refer­ring to a differ­ent fact than the base-op­ti­mizer. On the other hand, when the same met­ric is be­ing used by both par­ties, the class of pos­si­ble mis­takes does not in­clude what we’re not call­ing ter­minol­ogy is­sues.

  I think this also points to a fun­da­men­tal episte­molog­i­cal is­sue, one even broader than goal-rep­re­sen­ta­tion. It’s pos­si­ble that two mod­els dis­agree on rep­re­sen­ta­tion, but agree on all ob­ject level claims—think of us­ing differ­ent co­or­di­nate sys­tems. Be­cause ter­minol­ogy is­sues can cause mis­takes, I’d sug­gest that agents with non-shared world mod­els can only re­li­ably com­mu­ni­cate via ob­ject-level claims.

  The im­pli­ca­tion for AI al­ign­ment might be that we need AI to ei­ther fun­da­men­tally model the world the same way as hu­mans, or need to com­mu­ni­cate only via ob­ject-level goals and con­straints.

• Davidmanheim 19 Aug 2019 9:44 UTC

  2 points

  in reply to: Pattern's comment on: Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

  It seems like there’s a differ­ence be­tween the two cases. If I write a pro­gram to take the CRT, and then we both take it, and we both get the same score (and that isn’t a perfect score), be­cause it solved them the way I solve them, that doesn’t sound like mis­al­ign­ment.

  The mis­al­ign­ment here is be­tween you and the CRT, and re­flects your model be­ing mis­al­igned with the goal /​ re­al­ity. That’s why I’m call­ing it a prin­ci­pal al­ign­ment failure—even though it’s the pro­gram /​ mesa-op­ti­mizer that fails, the al­ign­ment failure is lo­cated in the prin­ci­pal, you /​ the base op­ti­mizer.

• Davidmanheim 19 Aug 2019 9:22 UTC

  2 points

  on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  Separate from my other com­ment, I want to ques­tion your as­sump­tion that we must worry about an AI-take­off that is ex­po­nen­tially bet­ter than hu­mans at ev­ery­thing, so that a very slight mis­al­ign­ment would be dis­as­trous. That seems pos­si­ble, per Eliezer’s Rocket Ex­am­ple, but is far from cer­tain.

  It seems likely that in­stead there are fun­da­men­tal limits on in­tel­li­gence (for a given ar­chi­tec­ture, at least) and while it is un­likely that the over­all limits are co­in­ci­den­tally the same as /​ near hu­man in­tel­li­gence, it seems plau­si­ble that the first su­per­hu­man AI sys­tem still plateaus some­where far short of in­finite op­ti­miza­tion power. If so, we only need to miti­gate well, in­stead of perfectly al­ign the AI to our goals.

• Davidmanheim 19 Aug 2019 9:15 UTC

  4 points

  on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  I’ll start by not­ing that I am in the strange (for me) po­si­tion of ar­gu­ing that some­one is too con­cerned about over-op­ti­miza­tion failures, rather than try­ing to con­vince some­one who is dis­mis­sive. But that said, I do think that the con­cern here, while real, is miti­gable in a va­ri­ety of ways.

  First, there is the pos­si­bil­ity of re­duc­ing op­ti­miza­tion pres­sure. One key con­tri­bu­tion here is Jes­sica Tay­lor’s Quan­tiliz­ers pa­per, which you note, that shows a way to build sys­tems that op­ti­mize but are not nearly as sub­ject to Good­hart’s curse. I think you are too dis­mis­sive. Similarly, you are dis­mis­sive of op­ti­miz­ing the tar­get di­rectly. I think that the episte­molog­i­cal is­sues you point to are pos­si­ble to miti­gate to the ex­tent that they won’t cause mis­al­ign­ment be­tween re­al­ity and an AI’s rep­re­sen­ta­tion of that re­al­ity. Once that is done, the re­main­ing is­sue is al­ign­ing “true” goals with the mea­sured goals, which is still hard, but cer­tainly not fun­da­men­tally im­pos­si­ble in the same way.

  Se­cond, you note that you don’t think we will solve al­ign­ment. I agree, be­cause I think that “al­ign­ment” pre­sup­poses a sin­gle co­her­ent ideal. If hu­man prefer­ences are di­verse, as it seems they are, we may find that al­ign­ment is im­pos­si­ble. This, how­ever, al­lows a very differ­ent ap­proach. This would op­ti­mize only when it finds Pareto-im­prove­ments across a set of sub-al­ign­ment met­rics or goals, to con­strain the pos­si­bil­ity of run­away op­ti­miza­tion. Even if al­ign­ment is pos­si­ble, it seems likely that we can spec­ify a set of di­verse goals /​ met­rics that are all al­igned with some hu­man goals, so that the sys­tem will be limited in its abil­ity to be mis­al­igned.

  Lastly, there is op­ti­miza­tion for a safe and very limited goal. If the goal is limited and spe­cific, and we find a way to min­i­mize side-effects, this seems like it could be fairly safe. For ex­am­ple, Or­a­cle AIs are an at­tempt to severely limit the goal. More broadly, how­ever, we might be able to build con­straints that work, so that it that can re­li­ably perform limited tasks (“put a straw­berry on a plate with­out pro­duc­ing any catas­trophic side-effects.”)

Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

• Davidmanheim 12 Aug 2019 1:34 UTC

  2 points

  AF

  on: In­terthe­o­retic util­ity com­par­i­son: examples

  This is very in­ter­est­ing—I hadn’t thought about util­ity ag­gre­ga­tion for a sin­gle agent be­fore, but it seems clearly im­por­tant now that it has been pointed out.

  I’m think­ing about this in the con­text of both the hu­man brain as an amalga­ma­tion of sub-agents, and or­ga­ni­za­tions as an amalga­ma­tion of in­di­vi­d­u­als. Note that we can treat or­ga­ni­za­tions as ra­tio­nally max­i­miz­ing some util­ity func­tion in the same way we can treat in­di­vi­d­u­als as do­ing so—but I think that for many or most vot­ing or de­ci­sion struc­tures, we should be able to rule out the claim that they are fol­low­ing any weighted com­bi­na­tion of nor­mal­ized util­ities of the agents in­volved in the sys­tem us­ing any in­terthe­o­retic com­par­i­son. This seems like a use­ful re­sult if we can prove it. (Alter­na­tively, it may be that cer­tain de­ci­sion rules map to spe­cific in­terthe­o­retic com­par­i­son rules, which would be even more in­ter­est­ing.)

• Davidmanheim 8 Aug 2019 15:04 UTC

  2 points

  in reply to: jacobjacob's comment on: How can we re­spond to info-cas­cades? [Info-cas­cade se­ries]

  1) It’s nei­ther noise nor rapid in­crease—it’s de­layed feed­back. Con­trol the­o­rists in en­g­ineer­ing have this as a re­ally clear, ba­sic re­sult, that de­layed feed­back is re­ally re­ally bad in var­i­ous ways. There are en­tire books on how to do it well—https://​​books.google.ch/​​books?id=Cy_wCAAAQBAJ&pg=PR9&lpg=PR9 - but do­ing it with­out us­ing these more com­plex tech­niques is bad.

  2) You ei­ther hire a con­trol the­o­rist, or (more prac­ti­cally) you avoid the cur­rent feed­back mechanism, and in­stead get peo­ple on the phone to talk about and un­der­stand what ev­ery­one needs, as op­posed to rely­ing on their de­layed feed­back in the form of nu­meric or­ders.

• Davidmanheim 8 Aug 2019 15:00 UTC

  2 points

  AF

  in reply to: abramdemski's comment on: What does Op­ti­miza­tion Mean, Again? (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 2)

  I think I agree about Eliezer’s defi­ni­tion, that it’s the­o­ret­i­cally cor­rect, but I definitely agree that I need to un­der­stand this bet­ter.

• Davidmanheim 7 Aug 2019 7:35 UTC

  8 points

  AF

  in reply to: abramdemski's comment on: Ap­ply­ing Overop­ti­miza­tion to Selec­tion vs. Con­trol (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 3)

  Thanks for the feed­back. I agree that in a con­trol sys­tem, any di­ver­gence be­tween in­tent and out­come is an al­ign­ment is­sue, and I agree that this makes overop­ti­miza­tion differ­ent in con­trol ver­sus se­lec­tion. De­spite the con­cep­tual con­fu­sion, I definitely think the con­nec­tions are worth not­ing—not only “wire­head­ing,” but the is­sues with mesa-op­ti­miz­ers. And I definitely think that causal failures are im­por­tant par­tic­u­larly in this con­text.

  But I strongly en­dorse how weak and fuzzy this is—which is a large part of why I wanted to try to de-con­fuse my­self. That’s the goal of this mini-se­quence, and I hope that do­ing so pub­li­cly in this way at least high­lights where the con­fu­sion is, even if I can’t suc­cess­fully de-con­fuse my­self, much less oth­ers. And if there are places where oth­ers are ma­te­ri­ally less con­fused than me and/​or you, I’d love for them to write re­sponses or their own ex­plain­ers on this.

• Davidmanheim 30 Jul 2019 14:43 UTC

  2 points

  AF

  in reply to: Charlie Steiner's comment on: What does Op­ti­miza­tion Mean, Again? (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 2)

  A cou­ple points.

  First, the rea­son why I wasn’t happy with en­tropy as a met­ric is be­cause it doesn’t al­low (straight­for­ward) com­par­i­son of differ­ent types of op­ti­miza­tion, as I dis­cussed. En­tropy of a prob­a­bil­ity dis­tri­bu­tion out­put isn’t com­pa­rable to the en­tropy over states that Eliezer defines, for ex­am­ple.

  Se­cond, I’m not sure false pos­i­tive and false nega­tive are the right con­cep­tual tools here. I can eas­ily show ex­am­ples of each—gra­di­ent de­scent can fail hor­ribly in many ways, and luck of spe­cific start­ing pa­ram­e­ters on spe­cific dis­tri­bu­tions can lead to un­rea­son­ably rapid con­ver­gence, but in both cases, it’s a re­la­tion­ship be­tween the al­gorithm and the space be­ing op­ti­mized.

• Davidmanheim 30 Jul 2019 5:02 UTC

  4 points

  AF

  on: Ap­ply­ing Overop­ti­miza­tion to Selec­tion vs. Con­trol (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 3)

  As always, con­struc­tive crit­i­cism on whether I’m still con­fused, or whether the points I’m mak­ing are clear, is wel­come!

Ap­ply­ing Overop­ti­miza­tion to Selec­tion vs. Con­trol (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 3)

What does Op­ti­miza­tion Mean, Again? (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 2)

• Davidmanheim 15 Jul 2019 15:29 UTC

  2 points

  AF

  in reply to: Stuart_Armstrong's comment on: In­differ­ence: mul­ti­ple changes, mul­ti­ple agents

  Yes, and this is a step in the right di­rec­tion, but as you noted in the writeup, it only ap­plies in a case where we’ve as­sumed away a num­ber of key prob­lems—among the most crit­i­cal of which seem to be:

  We have an as­sumed no­tion of op­ti­mal­ity, and I think an im­plicit as­sump­tion that the op­ti­mal point is unique, which seems to be needed to define re­ward—Abram Dem­ski has noted in an­other post that this is very prob­le­matic.

  We also need to know a sig­nifi­cant amount about both/​all agents, and com­pute ex­pec­ta­tions in or­der to de­sign any of their re­ward func­tions. That means fu­ture agents join­ing the sys­tem could break our agent’s in­differ­ence. (As an aside, I’m un­clear how we can be sure it is pos­si­ble to com­pute re­wards in a sta­ble way if their op­ti­mal policy can change based on the re­ward we’re com­put­ing.) If we can com­pute an­other agent’s re­ward func­tion when de­sign­ing our agent, how­ever, we can plau­si­bly hi­jack that agent.

  We also need a re­ward de­pend­ing on an ex­pec­ta­tion of ac­tions, which means we need coun­ter­fac­tu­als not only over sce­nar­ios, but over the way the other agent rea­sons. That’s a crit­i­cal is­sue I’m still try­ing to wrap my head around, be­cause it’s un­clear to me how a sys­tem can rea­son in those cases.

• Davidmanheim 14 Jul 2019 11:50 UTC

  2 points

  AF

  in reply to: Stuart_Armstrong's comment on: In­differ­ence: mul­ti­ple changes, mul­ti­ple agents

  The way the agents in­ter­act across in­ter­rup­tions seems to ex­actly par­allel in­ter­ac­tions be­tween agents where we de­sign for cor­rect be­hav­ior for agents sep­a­rately, and de­spite this, agents can cor­rupt the over­all de­sign by hi­jack­ing other agents. You say we need to de­sign for mu­tual in­differ­ence, but if we have a solu­tion that fixes the way they ex­ploit in­ter­rup­tion, it should also go quite a ways to­wards solv­ing the gen­er­al­ized is­sues with Good­hart-like ex­ploita­tion be­tween agents.

• Davidmanheim 10 Jul 2019 8:09 UTC

  2 points

  AF

  on: In­differ­ence: mul­ti­ple changes, mul­ti­ple agents

  This seems like an im­por­tant is­sue, but given the ex­am­ple, I’m also very in­ter­ested in how we can de­tect in­ter­ac­tions like this. Th­ese are effec­tively ex­am­ples of multi-party Good­hart effects, and the ex­am­ple you use is as­sumed to be “ob­vi­ous” and so a patch would be ob­vi­ously needed. This seems un­clear—we need to un­der­stand the strate­gic mo­tives to di­ag­nose what is hap­pen­ing, and given that we don’t have good ideas for ex­plain­abil­ity, I’m un­sure how in gen­eral to no­tice these effects to al­low patch­ing. (I have been work­ing on this and think­ing about it a bit, and don’t cur­rently have good ideas.)

• Davidmanheim 7 Jul 2019 10:38 UTC

  2 points

  AF

  on: Op­ti­miz­ing and Good­hart Effects Clar­ify­ing Thoughts—Parts 1 & 2

  Note: as I men­tioned in the post, I’d love feed­back of all types—for bet­ter clar­ity, crit­i­cism of my un­der­stand­ing, sug­ges­tions for how to con­tinue, and places the dis­cus­sion still seems con­fused.

• Davidmanheim 7 Jul 2019 10:37 UTC

  2 points

  in reply to: steve2152's comment on: Op­ti­miz­ing and Good­hart Effects Clar­ify­ing Thoughts—Parts 1 & 2

  I like that in­tu­ition over­all, but there’s a sense that adap­tive search does give far more re­s­olu­tion than grid search, but the anal­y­sis seems wrong; if I use gra­di­ent de­scent, by even­tual ac­cu­racy is the step size near the end, which gives far more pre­cise an­swers than “only” check­ing a grid of M^N points spaced equally.

• Davidmanheim 4 Jul 2019 8:09 UTC

  2 points

  AF

  in reply to: abramdemski's comment on: Con­cep­tual Prob­lems with UDT and Policy Selection

  Pretty sure we’re agree­ing here. I was origi­nally just sup­port­ing cousin_it’s claim, not claiming that Nash Equil­ibria are a use­ful-enough solu­tion con­cept. I was sim­ply not­ing that—while they are weaker than a use­ful-enough con­cept would be—they can show the is­sue with non-unique­ness clearly.

Re-in­tro­duc­ing Selec­tion vs Con­trol for Op­ti­miza­tion (Op­ti­miz­ing and Good­hart Effects—Clar­ify­ing Thoughts, Part 1)